Crazed Mule Productions

recording desktop with ffmpeg

2012-01-26T20:16:00.000-05:00

I had this idea that I'm going to start interviewing folks on different technical topics via Skype or Google Hangout. Normally, I would use Cinelerra to perform this task, but for some reason, the recordings in Cinelerra showed some irritating sound artifacts. Testing in audacity showed no such audio problems. I'll bring this up on the Cin mailing list soon.

FFmpeg to the Rescue
So with Cinelerra not working for desktop recordings, I needed an alternate way to record my desktop screen as well as the audio from the person being interviewed. After some searching a good deal of trial and error, I found from the link in the reference section at the bottom of the page, a very nice ffmpeg command. You can save to any format you want. The example below shows how to save the microphone audio (hw:0,0) in PCM format. The desktop video is brought in via the xllgrab format specifier :
ffmpeg -f alsa -i hw:0,0 -ac 2 -ar 48000 -acodec pcm_s16le -f x11grab -r 24 -s 1280x720 -i :0.0 -aspect 16:9 -vcodec libx264 -vpre lossless_ultrafast -threads 8 -y output.mov

I later tweaked the command to do a bit more, using pulse audio as input as well as saving out to an iPod compatible format:
ffmpeg -f alsa -i pulse -f x11grab -s 1280x720 -r 23.98 -i :0.0 -vcodec libx264 -acodec pcm_s16le -ab 128k -ar 48000 -ac 1 -b 2000k -bt 750k -refs 1 -deblockalpha 0 -deblockbeta 0 -subq 1 -me_range 21 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 2000k -bufsize 2M -cmp 1 -threads 8 output.mov

Errors
Not sure how yet to resolve these ffmpeg errors:
[alsa @ 0x16512a0] Estimating duration from bitrate, this may be inaccurate
Input #0, alsa, from 'hw:0,0':
Duration: N/A, start: 17479.447401, bitrate: N/A
Stream #0.0: Audio: pcm_s16le, 32000 Hz, 2 channels, s16, 1024 kb/s
[x11grab @ 0x1650240] device: :0.0 -> display: :0.0 x: 0 y: 0 width: 1280 height: 720
[x11grab @ 0x1650240] shared memory extension found
[x11grab @ 0x1650240] Estimating duration from bitrate, this may be inaccurate
Input #1, x11grab, from ':0.0':
Duration: N/A, start: 1327197224.425685, bitrate: 707788 kb/s
Stream #1.0: Video: rawvideo, bgra, 1280x720, 707788 kb/s, 24 tbr, 1000k tbn, 24 tbc
Incompatible pixel format 'bgra' for codec 'libx264', auto-selecting format 'yuv420p'

Along the way, I received an error:
"unknown field slave"

Finding a bit more about ALSA
This ended up being because I had edited my /etc/asound.conf file from the default. This corrupted my sound card settings. I was reacquainted with some debugging commands and places where "sound stuff" lives:
[sodo@computer etc]$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: U0x46d0x821 [USB Device 0x46d:0x821], device 0: USB Audio [USB Audio]
Subdevices: 0/1
Subdevice #0: subdevice #0
card 2: MobilePre [MobilePre], device 0: USB Audio [USB Audio]
Subdevices: 0/1
Subdevice #0: subdevice #0

[sodo@computer etc]$ amixer -c 2
Simple mixer control 'Speaker',0
Capabilities: pvolume pswitch pswitch-joined penum
Playback channels: Front Left - Front Right
Limits: Playback 0 - 31
Mono:
Front Left: Playback 31 [100%] [0.00dB] [on]
Front Right: Playback 31 [100%] [0.00dB] [on]
Simple mixer control 'PCM',0
Capabilities: pvolume pswitch pswitch-joined penum
Playback channels: Front Left - Front Right
Limits: Playback 0 - 31
Mono:
Front Left: Playback 25 [81%] [0.00dB] [on]
Front Right: Playback 25 [81%] [0.00dB] [on]
Simple mixer control 'PCM',1
Capabilities: volume volume-joined penum
Playback channels: Mono
Capture channels: Mono
Limits: 0 - 2
Mono: 2 [100%]
Simple mixer control 'Analog In',0
Capabilities: pvolume cvolume pswitch pswitch-joined cswitch cswitch-joined penum
Playback channels: Front Left - Front Right
Capture channels: Front Left - Front Right
Limits: Playback 0 - 31 Capture 0 - 31
Front Left: Playback 28 [90%] [6.00dB] [off] Capture 31 [100%] [12.00dB] [on]
Front Right: Playback 28 [90%] [6.00dB] [off] Capture 31 [100%] [12.00dB] [on]

[sodo@computer ~]$ cat /proc/asound/cards
0 [U0x46d0x821 ]: USB-Audio - USB Device 0x46d:0x821
USB Device 0x46d:0x821 at usb-0000:00:1d.7-6, high speed
1 [MobilePre ]: USB-Audio - MobilePre
M Audio MobilePre at usb-0000:00:1d.0-1, full speed

[sodo@computer etc]$ ls /dev/snd

by-id by-path controlC0 controlC2 pcmC0D0c pcmC2D0c pcmC2D0p seq timer

[sodo@computer etc]$ cat /etc/asound.conf

# Place your global alsa-lib configuration here...

@hooks [

{

func load

files [

"/etc/alsa/pulse-default.conf"

]

errors false

}

]

XWinInfo
Also, I learned about xwininfo, a cool command that shows you your desktop settings:
[sodo@computer videoConfTest]$ xwininfo
xwininfo: Window id: 0x320003b "Hak5 - Linux Screen Recording, Boxee Python Development and Qnext - YouTube - Google Chrome"

Absolute upper-left X: 0
Absolute upper-left Y: 53
Relative upper-left X: 0
Relative upper-left Y: 22
Width: 1280
Height: 971
Depth: 24
Visual: 0x21
Visual Class: TrueColor
Border width: 0
Class: InputOutput
Colormap: 0x20 (installed)
Bit Gravity State: NorthWestGravity
Window Gravity State: NorthWestGravity
Backing Store State: NotUseful
Save Under State: no
Map State: IsViewable
Override Redirect State: no
Corners: +0+53 -1280+53 -1280-0 +0-0
-geometry 1280x971+0-0

Finally, I installed Skype and dealt with resolving a bunch of dependency problems (as usual):

http://kezhong.wordpress.com/2011/09/02/installing-skype-on-fedora-15x86_64/

VideoCamming
And a simple way to trigger mplayer to open up your videocam:
mplayer tv:// -tv driver=v4l2 device=/dev/video0

Reference
http://www.linuxquestions.org/questions/linux-newbie-8/recording-desktop-with-ffmpeg-877580/
http://www.youtube.com/watch?v=mNz5Lrc06_s
http://forums.fedoraforum.org/showthread.php?t=256681
http://ffmpeg.org/ffprobe.html#pulse
http://alien.slackbook.org/blog/adding-an-alsa-software-pre-amp-to-fix-low-sound-levels/
http://forums.opensuse.org/english/get-technical-help-here/multimedia/471522-record-my-desktop-using-ffmpeg-post2433494.html
http://superuser.com/questions/251338/mencoder-capture-from-camera-and-microphone
http://ubuntuforums.org/showthread.php?t=1556038 (using jack)
http://wiki.linuxaudio.org/wiki/screencasttutorial (jack_capture)

Fedora 15 cinelerra install works, breaks MP2

2011-05-29T13:01:00.010-04:00

Fedora 15, x86-64 Cinelerra CV install from source
The Fedora 15, x86-64 installation of Cinelerra went well. I followed my previous Fedora 12, x86-64 installation instructions from my document here:
http://crazedmuleproductions.blogspot.com/2010/02/fedora-12-x86-64-cinelerra-install.html

The same instructions apply, along with the RPM build of FFMPEG for libfaac, the Apple Audio Codec.

Remove Video for Linux and IOMega Buzz drivers
Before I compiled the latest Cinelerra CV, I had to patch the codebase to remove Video for Linux and IOMega Buzz drivers, as they are not supported under the kernel (kernel 2.6.38) that Fedora 15 runs under. Here is the patch:
http://e.kevb.net/lurker/message/20110522.154000.ac0bc424.en.html

MPEG2 audio rendering broken
Once I got the code compiled, I tried reading and encoding video. An immediate problem I noticed is that MPEG Layer II encoding is broken. The rendering output gives no indication that there is a problem:
Input File : 'stdin' 48.0 kHz Output File: '/home/sfrase/videos/stormpigs/2010_12_28/testmp2monty_384.m2a' 384 kbps MPEG-1 Layer II j-stereo Psy model 1 [De-emph:Off Copyright:No Original:No CRC:Off] [Padding:Normal Byte-swap:Off Chanswap:Off DAB:Off] ATH adjustment 0.000000 -------------------------------------------- encode_init: using tablenum 0 with sblimit 27 Hit end of audio data Avg slots/frame = 1152.000; b/smp = 8.00; bitrate = 384.000 kbps Render::run: Session finished.

However, both mplayer and ffplay spit out similar messages that the MP2 has a header missing:
mplayer
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, floatle, 384.0 kbit/12.50% (ratio: 48000->384000)
Selected audio codec: [ffmp2float] afm: ffmpeg (FFmpeg MPEG layer-1 and layer-2 audio)
==========================================================================
AO: [pulse] 48000Hz 2ch floatle (4 bytes per sample)
Video: no video
Starting playback...
[mp2float @ 0x327153c6a0]Header missing
[mp2float @ 0x327153c6a0]Header missing
A: 0.0 (00.0) of 342.7 (05:42.6) ??,?%

FFplay
Input #0, mp3, from 'testmp2monty_384.m2a':
Duration: 00:05:42.67, start: 0.000000, bitrate: 384 kb/s
Stream #0.0: Audio: mp2, 48000 Hz, stereo, s16, 384 kb/s
[mp2 @ 0x7f27c8000af0] Header missing

Playing back the audio is garbled. This isn't a deal breaker, but it is a little upsetting. I thought I might escape it this time, but I guess I will have to bite the bullet and compatibility test all my Cinelerra file formats with Fedora 15.

Compiling Cinelerra Monty from source
Since Cinelerra CV worked at a basic level, I wanted to compile my preferred Cinelerra distro, Cinelerra Monty:
http://crazedmuleproductions.blogspot.com/2010/07/green-beret-parachutes-in-to-help.html

However, this codebase also needed the updates to remove Video for Linux and IOMega Buzz drivers. Since Monty's code base differs from CV, I had to manually patch the code myself using the instructions above. Once that was complete, I was able to successfully compile CinMonty.

References
http://bugs.cinelerra.org/ticket/949
Nicola's patch
Creating a patch using Diff

two-pass encoding in Cinelerra

2010-10-26T20:23:00.011-04:00

So I was creating a video to showcase my cousin's ambient/electroacoustic music and needed the best output possible. I put together the project with scraps of 1080P video from my Canon 5D that I had lying around.

Normally, I would do my edits and then post an MPEG-TS to Vimeo. However, as Florian had called out recently, I found that portions of the final video were slightly underwhelming in quality. Specifically, some timelapses in the video looked grainy. This was because the MPEG2 render parameters I had been using in the above link were giving me less than best quality. So my solution was to use a two pass encode from Cinelerra via the YUV4MPEG stream capability into an MPEG4.

Easier said then done.

There are three steps:
1) export the audio
2) perform the first pass encoding into /dev/null
3) perform the second pass encoding with the audio file into the final file

Export the Audio
As I was having some sync issues, I pushed the audio a little further down the timeline using a negative nudge value.

First Pass Encode
Cinelerra seems to be very sensitive to any garbage in the command line sent to the YUV4MPEG stream. Look at this output:

Render::run 10
trying popen(ffmpeg -threads 8 -y -i - -an -v 1 -vcodec libx264 -aspect 1.7777 -b 9000k -bt 7775k -refs 1 -deblockalpha 0 -deblockbeta 0 -subq 1 -me_range 21 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 10000k -bufsize 2M -cmp 1 -f mp4 -pass 1 /dev/null� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ���)
FFmpeg version 0.6-rpmfusion, Copyright (c) 2000-2010 the FFmpeg developers
  built on Sep 23 2010 18:55:52 with gcc 4.4.4 20100630 (Red Hat 4.4.4-10)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --extra-version=rpmfusion --enable-bzlib --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-nonfree --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avfilter-lavf --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
[yuv4mpegpipe @ 0xf0a5f0]Estimating duration from bitrate, this may be inaccurate
Input #0, yuv4mpegpipe, from 'pipe:':
  Duration: N/A, bitrate: N/A
    Stream #0.0: Video: rawvideo, yuv420p, 1920x1088, PAR 1:1 DAR 30:17, 30 tbr, 30 tbn, 30 tbc
/dev/null� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ��� ���: File name too long
Received sigpipe
Received sigpipe
Render::run: Session finished.

That is just damn ugly. It looks like some crap got pasted into the command string. Even though I couldn't see anything at the end of the line in the YUV4MPEG stream, I hit the delete key a few times. The method cleaned out whatever invisible control characters got into the command.

My first pass encoding string was this:

ffmpeg -threads 8 -y -i - -an -v 1 -vcodec libx264 -aspect 1.7777 -b 9000k -bt 7775k -refs 1 -deblockalpha 0 -deblockbeta 0 -subq 1 -me_range 21 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 10000k -bufsize 2M -cmp 1 -f mp4 -pass 1 /dev/null

Note that
1) the audio file is not merged at this first pass step.
2) there is no output file in the first pass encode, the output goes directly to /dev/null

Second Pass Encode
The second pass encode string is where I add my audio file and my output. In the second pass encode string below, note the appearance of the audio file path and the % at the end of the YUV4MPEG stream indicating the output file:

ffmpeg -threads 8 -y -i - -i /mnt/videos/projects/rickr/final.mp3 -v 1 -vcodec libx264 -aspect 1.7777 -b 9000k -bt 7775k -refs 1 -deblockalpha 0 -deblockbeta 0 -subq 1 -me_range 21 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 10000k -bufsize 2M -acodec libfaac -ab 256k -ar 44000 -ac 2 -cmp 1 -f mp4 -pass 2 %

Note that I could have done these commands outside of Cinelerra if I had output the video to some intermediate file format, usually Quicktime for Linux with JPEG or MJPEG compression for best results.

Every project I do leads me to find some bit of minutiae that sticks in my brain. This time, the minutiae is an FFMPEG bug wherein the parameter to make adjustments to the audio sync does not work for MPEG4 files:
https://roundup.ffmpeg.org/issue359

Final Result
All-in-all, I think this pain gave some very nice results. Let me know what you think:

A Little Time Away from crazed mule on Vimeo.

Music By Rick Rascati from his album Cinemusica
http://myspace.com/rickrmusic
Shot with a Canon 5D (90%), other JVC HD10U

Another one bites the dust. Have a good one, folks.
the mule

Reference
http://howto-pages.org/ffmpeg/

hugin, open source software for panoramas

2010-10-04T18:37:00.004-04:00

Hugin..very nice software for panoramas.

I stitched together a few nighttime photos from north carolina with this result:

I'm sure it could be better..have to work on the technique.

My flickr set:
http://www.flickr.com/photos/frases/sets/72157625086498978/show/

Reference
http://hugin.sourceforge.net/
http://lifehacker.com/378490/stitch-photos-into-panoramas-with-free-software

finding mem leaks with valgrind

2010-09-23T23:35:00.008-04:00

I had been playing around with CinMonty for a couple weeks now and noticed a fairly big memory leak when I played back or rendered MPEG-PS or the AVC files from my Canon 5D(actually, H264/PCM audio). It must be stated that though I took some programming in school, I'm no C programmer. Handy with the shell, but not a C programmer. In any case, I found it interesting to try and find a memory leak in Cinelerra Monty using valgrind, a profiler/instrumenter/error detector of C programs.

Starting Valgrind
Valgrind is executed at the command line with the name of the program that valgrind will analyze. You can start valgrind with plenty of options, but I started with a few common arguments:
-check for leaks
-log to a file
-log unlimited errors

The command line looks like this:
[mule@ogre 2010_09_22]$ valgrind --leak-check=full --log-file=memLeakCinMonty.txt --error-limit=no cinelerra
Cinelerra 2.1CV
xiphmont GIT::a515d4be161fde852b72471f631782debf5d32f5
(C) 2006 Heroine Virtual Ltd.
(C) 2006-2010 The CinelerraCV Community
Internal ffmpeg 0.6+fixes
Compiled on Sat Sep 18 14:04:36 EDT 2010

Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra. FFMPEG::init_picture failed
FFMPEG::init_picture failed
FFMPEG::init_picture failed

I'm not sure what the init_picture failed error messages are, but they occur when I playback or render MPEG-PS or H264 video. This may have something to do with the memory leak.

When running cinlerra under valgrind, the performance of Cinelerra grinds to a halt. Perhaps that's why Julian Seward (the original designer and author of Valgrind) called it valGRIND. However, performance is not so bad that you can't get usable data out of the program. For instance, valgrind shows me the following interesting section..an overlap in memcpy:
==23140== Thread 36:
==23140== Source and destination overlap in memcpy(0x3ce5e1b0, 0x3ce7e1b0, 526848) ==23140== at 0x4A06A3A: memcpy (mc_replace_strmem.c:497)
==23140== by 0x569092: FileBase::update_pcm_history(long) (filebase.C:105)
==23140== by 0x5872F5: FileFFMPEG::read_samples(double*, long) (fileffmpeg.C:648) ==23140== by 0x56ACAA: File::read_samples(double*, long, long, float*) (file.C:1042) ==23140== by 0x500CAE: AModule::render(double*, long, int, int, int, int) (amodule.C:258)
==23140== by 0x658543: VirtualANode::read_data(double*, long, long, long) (virtualanode.C:161)
==23140== by 0x658820: VirtualANode::render_as_module(double**, double*, long, long, long) (virtualanode.C:238)
==23140== by 0x65897A: VirtualANode::render(double*, long, long, long) (virtualanode.C:178)
==23140== by 0x6576F7: VirtualAConsole::process_buffer(long, long, int, long) (virtualaconsole.C:134)
==23140== by 0x502B06: ARender::process_buffer(long, long) (arender.C:232)
==23140== by 0x502969: ARender::run() (arender.C:325)
==23140== by 0x51B4425: Thread::entrypoint(void*) (thread.C:69)

It's good that valgrind found something. Now to fix it! I've handed this info off to Monty. Hopefully he will be able to replicate my problem and fix it.
:)
the mule

making sure opengl is available

2010-09-10T16:47:00.005-04:00

This troubleshooting is listed in this post:
https://init.linpro.no/pipermail/skolelinux.no/cinelerra/2010-January/016493.html

But I thought I'd repost here so that I always have this information at hand:

I have a GeForce 8800GT card installed in my box. I see glxinfo says I have OpenGL:
[sfr...@ogre my_cinelerra]$ glxinfo | head -20
name of display: :0.0
display: :0 screen: 0
direct rendering: Yes
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4

I see glxgears points to the libGL.so.1 in /usr/lib64/nvidia:
[sfr...@ogre my_cinelerra]$ ldd `which glxgears`
linux-vdso.so.1 => (0x00007fff85dff000)
libGL.so.1 => /usr/lib64/nvidia/libGL.so.1 (0x00007f791fa75000)
libm.so.6 => /lib64/libm.so.6 (0x0000003f1bc00000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003f1e800000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003f1c400000)
libc.so.6 => /lib64/libc.so.6 (0x0000003f1b800000)

libGLcore.so.1 => /usr/lib64/nvidia/libGLcore.so.1 (0x00007f791e5a3000) libnvidia-tls.so.1 => /usr/lib64/nvidia/tls/libnvidia-tls.so.1 (0x00007f791e4a1000)

libXext.so.6 => /usr/lib64/libXext.so.6 (0x0000003f1ec00000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003f1c000000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000003f1e400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003f1b400000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003f1e000000

I see that the libGL.so.1 in that directory has the appropriate OpenGL hooks:
[sfr...@ogre usr]$ strings -a /usr/lib64/nvidia/libGL.so.1 | grep glDeleteShader
glDeleteShader
[sfr...@ogre usr]$ strings -a /usr/lib64/nvidia/libGL.so.1 | grep glUseProgram
glUseProgram
glUseProgramObjectARB

I've tried manually pointing configure.in to the /usr/lib64/nvidia directory:
AC_CHECK_LIB([GL], [glUseProgram],
[OPENGL_LIBS="-lGL"; libGL=yes],
# On SUSE/OpenSUSE, NVidia places the OpenGL 2.0 capable library in
/usr/X11R6/lib
# but it doesn't place a libGL.so there, so the linker won't pick it up
# we have to use the explicit libGL.so.1 path.
save_LIBS="$LIBS"
for l in /usr/lib64/nvidia /usr/X11R6/lib /usr/X11R6/lib64; do
LIBS="$l/libGL.so.1"
AC_MSG_CHECKING(for glUseProgram in $l/libGL.so.1)
AC_TRY_LINK([],[extern int glUseProgram();
glUseProgram();],
[OPENGL_LIBS="$l/libGL.so.1";
libGL=yes],[libGL=no])
AC_MSG_RESULT([$libGL])
test $libGL = yes && break
done
LIBS="$save_LIBS"

In the end, even though ./configure did not recognize that I had openGL properly installed, I had to explicitly enable opengl on my ./configure line:
./configure --enable-opengl

Who knew?
da mule

building CinCV from source, Ubuntu 32-bit

2010-08-06T09:17:00.008-04:00

Normally, I am a Fedora man, but I was helping a friend install Cinelerra on Ubuntu Lucid 10.04 32-bit. I installed Lucid in a VMware virtual machine on VMware Server. Here are some notes that may be helpful for someone. The instructions assume a base installation. The base installation VM file size is about 2.7GB. With my updates, it becomes 4GB.

echo "1: update manually with synaptic package manager"
I don't know the command line for this, just do via GUI

echo "2: install dependencies for source"
sudo apt-get install g++ git-core libtool automake nasm mplayer xorg-dev ffmpeg libasound2-dev libogg-dev libvorbis-dev libtheora-dev libopenexr-dev libdv4-dev libpng12-dev libjpeg62-dev libx264-dev uuid-dev mjpegtools libmjpegtools-dev libfftw3-dev liba52-0.7.4-dev libmp3lame0 libmp3lame-dev libsndfile1-dev libfaac-dev libfaad-dev libesd0-dev libavc1394-dev libraw1394-dev libiec61883-dev libtiff4-dev libxxf86vm-dev libglu1-mesa-dev libogg-dev libvorbis-dev libopenexr-dev libpng12-dev libjpeg62-dev mjpegtools libmp3lame0 libquicktime-dev libqt4-dev libpostproc-dev libavformat-dev libavutil-dev libmad0-dev libhal-dev libdbus-1-dev libgcrypt11-dev libfribidi-dev

echo "3: touch language files not present"
cd $HOME/my_cinelerra/po;touch de.gmo es.gmo eu.gmo fr.gmo it.gmo pt_BR.gmo ru.gmo sl.gmo

echo "4: chmod on stamp-po for make install"
cd $HOME/my_cinelerra/po;sudo chmod 777 stamp-po

echo "5: run ldconfig to get around libquicktimehv error"
sudo ldconfig

echo "6: add kernel parameter for max shared memory"
sudo su; echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf

Compile from source (executed within source code directory):
1) ./autogen.sh
2) ./configure
4) make
5) make install

Compile with Monty's changes
After verifying a successful CinCV compile, you'd want to load up Monty's changes and then do:
1) ./autogen.sh
2) ./configure
3) make clean
4) make
5) make install

da mule

Canon 5D conversion for xiphmont's git

2010-07-24T15:07:00.014-04:00

As Monty has been working hard to make Cinelerra more input friendly with his FFmpeg loader fixes, I've been trying to help out as a QA resource. One of my main concerns has been getting my Canon 5D video into Cinelerra. As documented in the IRC log, my custom Canon conversion (ffmpeg stream -> mpeg2enc (output: .m2v/.m2a) -> mplex (output MPEG-PS) -> ffmpeg (output MPEG-TS) was full of invalid timestamps.

In addition, with some recent security updates to my Fedora system just today, it looks like my previous Canon conversion script had broken. I get the dread error:

**ERROR: [mpeg2enc] display_horizontal_size must be in range 0...16383

So I was doubly screwed. Back to the drawing board..er, mailing list. Following up on mjpegtools' Bernhard Prachinger's advice here, I first try to play a test file using a pipe from ffmpeg, like so:
ffmpeg -i mvi_2655.mov -threads 7 -s 1920x1088 -f yuv4mpegpipe - | yuvplay

The video stream plays back with no visible errors, outside of a warning about the video stream frame rate differing from the container frame rate. Here are the top few lines of the output:
Seems stream 0 codec frame rate differs from container frame rate: 48000.00 (48000/1) -> 23.98 (24000/1001)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'mvi_2655.mov':
Duration: 00:00:06.92, start: 0.000000, bitrate: 45836 kb/s
Stream #0.0(eng): Video: h264, yuv420p, 1920x1088, 44296 kb/s, 23.98 tbr, 24k tbn, 48k tbc
Stream #0.1(eng): Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Metadata
major_brand : qt
minor_version : 537331968
compatible_brands: qt CAEP
Output #0, yuv4mpegpipe, to 'pipe:':
Stream #0.0(eng): Video: rawvideo, yuv420p, 920x108, q=2-31, 200 kb/s, 90k tbn, 23.98 tbc
Stream mapping:
Stream #0.0 -> #0.0
Press [q] to stop encoding
INFO: [yuvplay] Playing frame 0000 - 0:00:00.00
INFO: [yuvplay] Playing frame 0001 - 0:00:00.01
INFO: [yuvplay] Playing frame 0002 - 0:00:00.02
INFO: [yuvplay] Playing frame 0003 - 0:00:00.03
INFO: [yuvplay] Playing frame 0004 - 0:00:00.04
INFO: [yuvplay] Playing frame 0005 - 0:00:00.05
INFO: [yuvplay] Playing frame 0006 - 0:00:00.06
INFO: [yuvplay] Playing frame 0007 - 0:00:00.077 bitrate=752028.9kbits/s

So, streaming from FFmpeg to a y4m stream works. I then take that y4m stream and convert it to an mpeg2video for use in Cinelerra, again using Bernhard's simplified mpeg2enc pipe adjusted for aspect ratio and higher bitrate:
ffmpeg -i mvi_2655.mov -threads 7 -s 1920x1088 -f yuv4mpegpipe - | mpeg2enc --verbose 2 --format 3 --aspect 3 --video-bitrate 24000 --no-constraints --video-buffer 448 -o mvi_2655.m2v

Looking at the output (--verbose 2), there don't seem to be any major errors here:
Seems stream 0 codec frame rate differs from container frame rate: 48000.00 (48000/1) -> 23.98 (24000/1001)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'mvi_2655.mov':
Duration: 00:00:06.92, start: 0.000000, bitrate: 45836 kb/s
Stream #0.0(eng): Video: h264, yuv420p, 1920x1088, 44296 kb/s, 23.98 tbr, 24k tbn, 48k tbc
Stream #0.1(eng): Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Metadata
major_brand : qt
minor_version : 537331968
compatible_brands: qt CAEP
Output #0, yuv4mpegpipe, to 'pipe:':
Stream #0.0(eng): Video: rawvideo, yuv420p, 920x108, q=2-31, 200 kb/s, 90k tbn, 23.98 tbc
Stream mapping:
Stream #0.0 -> #0.0
Press [q] to stop encoding
INFO: [mpeg2enc] Selecting Generic MPEG2 output profile
INFO: [mpeg2enc] Assuming norm NTSC
INFO: [mpeg2enc] Progressive input - selecting progressive encoding.
INFO: [mpeg2enc] Encoding MPEG-2 video to mvi_2655.m2v
INFO: [mpeg2enc] Horizontal size: 920 pel
INFO: [mpeg2enc] Vertical size: 108 pel
INFO: [mpeg2enc] Aspect ratio code: 3 = 16:9 display
INFO: [mpeg2enc] Frame rate code: 1 = 24000.0/1001.0 (NTSC 3:2 pulldown converted FILM)
INFO: [mpeg2enc] Bitrate: 24000 KBit/s
INFO: [mpeg2enc] Quality factor: 8 (Quantisation = 9) (1=best, 31=worst)
INFO: [mpeg2enc] Field order for input: none/progressive
INFO: [mpeg2enc] Sequence unlimited length
INFO: [mpeg2enc] Search radius: 16
INFO: [mpeg2enc] DualPrime: no
INFO: [mpeg2enc] Using one-pass rate controller
INFO: [mpeg2enc] GOP SIZE RANGE 7 TO 15
INFO: [mpeg2enc] Setting colour/gamma parameters to "NTSC"
INFO: [mpeg2enc] Progressive format frames = 1
INFO: [mpeg2enc] Using default unmodified quantization matrices
INFO: [mpeg2enc] SETTING MMX and MMX for QUANTIZER!
--DEBUG: [mpeg2enc] PAR = 7

I then take the output of the audio conversion:
ffmpeg -i mvi_2655.mov -acodec mp2 -ab 384k -ar 48000 -ac 2 test.m2a

Output:
Seems stream 0 codec frame rate differs from container frame rate: 48000.00 (48000/1) -> 23.98 (24000/1001)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'mvi_2655.mov':
Duration: 00:00:06.92, start: 0.000000, bitrate: 45836 kb/s
Stream #0.0(eng): Video: h264, yuv420p, 1920x1088, 44296 kb/s, 23.98 tbr, 24k tbn, 48k tbc
Stream #0.1(eng): Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Metadata
major_brand : qt
minor_version : 537331968
compatible_brands: qt CAEP
Output #0, mp2, to 'mvi_2655.m2a':
Stream #0.0(eng): Audio: mp2, 48000 Hz, 2 channels, s16, 384 kb/s
Stream mapping:
Stream #0.1 -> #0.0
Press [q] to stop encoding
size= 325kB time=6.94 bitrate= 384.0kbits/s
video:0kB audio:325kB global headers:0kB muxing overhead 0.000000%

and mux both the video and audio streams to an MPEG-PS in mplex:
mplex -f 3 -b 2000 mvi_2655.m2a mvi_2655.m2v -o mvi_2655.ps

Output:
INFO: [mplex] mplex version 1.9.0 (2.2.7 $Date: 2006/02/01 22:23:01 $)
INFO: [mplex] File mvi_2655.m2a looks like an MPEG Audio stream.
INFO: [mplex] File mvi_2655.m2v looks like an MPEG Video stream.
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting generic MPEG2 output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: Audio stream c0 (mvi_2655.m2a)
INFO: [mplex] MPEG AUDIO STREAM: c0
INFO: [mplex] Audio version : 1.0
INFO: [mplex] Layer : 2
INFO: [mplex] CRC checksums : no
INFO: [mplex] Bit rate : 49152 bytes/sec (384 kbit/sec)
INFO: [mplex] Frequency : 48000 Hz
INFO: [mplex] Mode : 0 stereo
INFO: [mplex] Mode extension : 0
INFO: [mplex] Copyright bit : 0 no copyright
INFO: [mplex] Original/Copy : 1 original
INFO: [mplex] Emphasis : 0 none
INFO: [mplex] Scanning for header info: Video stream e0 (mvi_2655.m2v)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 1920
INFO: [mplex] Frame height : 1088
INFO: [mplex] Aspect ratio : 16:9 display
INFO: [mplex] Picture rate : 23.976 frames/sec
INFO: [mplex] Bit rate : 24000000 bits/sec
INFO: [mplex] Vbv buffer size : 229376 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 24894496
INFO: [mplex] Setting best-guess data rate.
INFO: [mplex] Run-in delay = 30030 Video delay = 30030 Audio delay = 33783
INFO: [mplex] New sequence commences...
INFO: [mplex] Audio c0: buf= 0 frame=000000 sector=00000000
INFO: [mplex] Video e0: buf= 0 frame=000000 sector=00000000
INFO: [mplex] Scanned to end AU 165
INFO: [mplex] STREAM e0 completed
INFO: [mplex] STREAM c0 completed
INFO: [mplex] Multiplex completion at SCR=653806.
INFO: [mplex] Audio c0: completed
INFO: [mplex] Video e0: completed
INFO: [mplex] AUDIO_STATISTICS: c0
INFO: [mplex] Audio stream length 332928 bytes.
INFO: [mplex] Syncwords : 289
INFO: [mplex] Frames : 289 padded
INFO: [mplex] Frames : 0 unpadded
INFO: [mplex] BUFFERING min 189 Buf max 1162
INFO: [mplex] VIDEO_STATISTICS: e0
INFO: [mplex] Video Stream length: 13871991 bytes
INFO: [mplex] Sequence headers: 1
INFO: [mplex] Sequence ends : 1
INFO: [mplex] No. Pictures : 166
INFO: [mplex] No. Groups : 12
INFO: [mplex] No. I Frames : 12 avg. size162167 bytes
INFO: [mplex] No. P Frames : 154 avg. size 77441 bytes
INFO: [mplex] No. B Frames : 0 avg. size 0 bytes
INFO: [mplex] Average bit-rate : 16028800 bits/sec
INFO: [mplex] Peak bit-rate : 21470400 bits/sec
INFO: [mplex] BUFFERING min 85 Buf max 1944774
INFO: [mplex] MUX STATUS: no under-runs detected

This gives me a good program stream file that is suitable for editing in Cinelerra using Monty's changes. Update 2010/09/13: Florian Cramer advised me that the video render string does not give the highest quality output. I'm still working on improvements to the string that will render at a higher quality.

Moral of the story: never update your system.
;)
the mule

Reference
http://man-wiki.net/index.php/1:mplex
Minor headache with nvidia drivers after kernel upgrade resolved
Plus another pulseaudio fuck job.."pulseaudio module-alsa-card file not found"
- had to reinstall pulseaudio and then execute "alsamixer -c0" to fix

Green Beret parachutes in to help Cinelerra

2010-07-16T14:14:00.038-04:00

Unbelievably, just in the past week, the Cinelerra CV community (http://cvs.cinelerra.org/) has had a heavyweight developer parachute in and start fixing stuff all over the place. The guy is Chris "Monty" Montgomery who developed the Ogg multimedia container and Vorbis audio formats (http://en.wikipedia.org/wiki/Chris_Montgomery). He is a developer currently working for Red Hat.

Monty, You're The Man!
His primary effort has been to yank out the kludgy and incompatible file loader in Cinelerra and replace it using good old FFmpeg. FFmpeg has problems of its own, but a helluva lot less problems than the Cinelerra loader. This is a task that the original developer of Cinelerra, Heroine Warrior (http://www.heroinewarrior.com/), has wisely done with the 4.1 version of Cinelerra. Much props to Father HV, for without him, this blog and Open Source video editing would be very different, if exist at all.

Monty is patching the version of FFmpeg that comes with the Cinelerra CV source code (http://cinelerra.org/getting_cinelerra.php). In Cinelerra CV's source code tree, FFmpeg is found under quicktime/ffmpeg. This means that once you compile Cinelerra from source code and add Monty's code fixes, files like the Canon 5D/7D H264 AVC files (the 1080P files spat out of the Canons') will load natively in Cinelerra.

AVC, oh AVC, Why Are You So Stubborn?
Problem here is that you can't really edit with AVC files because the decoding of them is so bloody CPU intensive. AVC playback in Cinelerra on my dual, quad core Dell SC1430 with 10GB RAM is 3-4fps. This is unacceptable for previewing edits. There is a workaround to this such that if you really wanted to edit AVC, you'd need to have background rendering enabled, make your edits and then wait for the background render to finish. But that bg render takes a long time too. You could speed this up to by creating a ramdisk (http://www.techanswerguy.com/2009/02/creating-ramdisk-in-linux.html) and setting the background render to use that ramdisk. But that's a whole other task.

Transcoding Canon 5D/7D Files, as usual
Thus, we're still left with the task of transcoding Canon 5D/7D files into a more compatible format. There are two solutions:
1) get my varied MPEG-TS conversion magic to work for you (http://crazedmuleproductions.blogspot.com/2009/02/dark-of-winter-has-me-in-its-grasp.html)
2) convert the 5D/7D files using FFmpeg into another format that works since FFmpeg is now Cinelerra's loader (with Monty's additions)

Both require research/work, but I did a quick conversion that I tested in Cinelerra with Monty's code additions:
ffmpeg -i [inputFileFromCanon] -acodec mp2 -ar 48000 -ab 384k -ac 2 -vcodec mpeg2video -qscale 1 [outputFile]

The audio format is mpeg, layer II and the video format is mpeg2video. This loads in Cinelerra and is fast to edit with. Once edited, render out from Cinelerra as usual.

How to Compile Cinelerra for Ubuntu and Fedora users
So Monty's additions require compiling from source. I'll provide some quick links here. For Ubuntu folks, Raffaella Traniello has put a nice post together on how to compile Cinelerra in Ubuntu:
http://www.g-raffa.eu/Cinelerra/HOWTO/compilation.html

For Fedora users, I have a post here:
http://crazedmuleproductions.blogspot.com/2010/02/fedora-12-x86-64-cinelerra-install.html

Get Monty's Additions Working
Once you get the source code compiled, install it (make install) and just test that Cinelerra works at a basic level. Once you've tested that your compile has worked, move onto apply Monty's code additions:
- remove your installed Cinelerra (make uninstall)
- download Monty's additions at http://people.xiph.org/~xiphmont/cinelerra/patches/
- apply each of them to the Cinelerra source code tree one at a time (git apply [patch name])
or join in Monty's git branch:
1) git remote add xiphmont git://git.xiph.org/users/xiphmont/cinelerraCV.git
2) git remote update
- compile Cinelerra, this time with all Monty's additions (make clean;make;make install)

After starting CinCV with Monty's additions, you should see the startup info change:
[mule@ogre my_cinelerra]$ cinelerra
Cinelerra 2.1CVxiphmont
GIT::8c01c13bbb21d30aff2733454a49fbfe641e51ca
(C) 2006 Heroine Virtual Ltd.
(C) 2006-2010 The CinelerraCV Community
Internal ffmpeg 0.6+fixes
Compiled on Fri Jul 16 20:40:38 EDT 2010

As well, you should see that FFmpeg becomes the default loader for videos in you goto the Resources window, right-click on a video and click "Info". You should see FFMPEG there under File Format:

A Few Git Notes..
While I'm at it..

View current git branches
[mule@ogre my_cinelerra]$ git branch
* master

Add Monty's remote branch
[mule@ogre my_cinelerra]$ git remote add xiphmont git://git.xiph.org/users/xiphmont/cinelerraCV.git

Update the local copy of Monty's remote branch
[mule@ogre my_cinelerra]$ git remote update
Fetching origin
Fetching xiphmont
remote: Counting objects: 2754, done.
remote: Compressing objects: 100% (1904/1904), done.
remote: Total 1981 (delta 997), reused 0 (delta 0)
Receiving objects: 100% (1981/1981), 2.77 MiB 1.07 MiB/s, done.
Resolving deltas: 100% (997/997), completed with 711 local objects.
From git://git.xiph.org/users/xiphmont/cinelerraCV
* [new branch] master -> xiphmont/master

Check that Monty's branch has been added
[mule@ogre my_cinelerra]$ git branch
* master
xiphmont

I then was able to switch branches
[mule@ogre my_cinelerra]$ git checkout -b xiphmont xiphmont/master
Branch xiphmont set up to track remote branch master from xiphmont.
Switched to a new branch 'xiphmont'

"git branch" shows that I have switched to Monty's branch
[mule@ogre my_cinelerra]$ git branch
master
* xiphmont

Now I can switch easily between branches (Monty's and CinelerraCV)
[mule@ogre my_cinelerra]$ git checkout master
Switched to branch 'master'

Switching to Monty's branch
[mule@ogre my_cinelerra]$ git checkout xiphmont
Switched to branch 'xiphmont'

Show the current status of the git code repositories
[sfrase@ogre my_cinelerra]$ git status | head -6
# On branch xiphmont
# Untracked files:
# (use "git add ..." to include in what will be committed)
#
# cinelerra-cvs-current.spec
# cinelerra/.deps/
>

View the latest change log
[mule@ogre my_cinelerra]$ git log xiphmont/master | head -6
commit 8c01c13bbb21d30aff2733454a49fbfe641e51ca
Author: Monty
Date: Fri Jul 16 07:30:10 2010 -0400

iAlter/extend versioning to avoid any confusion between this temporarily
forked version and the main CinelerraCV repo

Pull in the latest code updates:
[mule@ogre my_cinelerra]$ git pull

Troubleshooting
I tried to switch branches and got this error
[mule@ogre my_cinelerra]$ git checkout -b xiphmont xiphmont/master
error: You have local changes to 'po/de.po'; cannot switch branches.

I did a hard reset to clear the above error
[mule@ogre my_cinelerra]$ git reset --hard HEAD
HEAD is now at d95887f Pass --libdir given to configure on to ffmpeg configure.

Last resort for repos not working, a destructive clean:
[mule@ogre my_cinelerra]$ git clean -dfx

Update 2010/07/16
Looks like the YUV4MPEG render stream is broken..this simple render pipe outputs a file unreadable in FF
ffmpeg -f yuv4mpegpipe -i - -y -target dvd -f mpeg2video %

I talked to Monty and he will fix..something about colorspace converter.
*** end update ***

Note that you have to compile Cinelerra and do a full reinstall, as the Cinelerra "make" process hard links the installation directory into the cinelerra executable.

So I hope Monty stays with us as long as possible. Thanks man! Hope springs eternal!
the mule

References
http://www.itworld.com/open-source/115063/breaking-open-video-frontier-despite-mpeg-la

extundelete to the rescue!

2010-06-05T12:08:00.004-04:00

So..in usual fashion, I had a minor foul up today. I was overzealous in my desire to reclaim disk space and zapped some primary source vids off of my ext4 partition. Ooops. Thankfully, I found extundelete, a program that scans ext3 and ext4 filesystem journals and recovers files from those journals:
http://extundelete.sourceforge.net/

Process
Here's what I did to recover the files.
1) In order to prevent the deleted files being overwritten, I immediately stopped whatever work I was doing and unmounted the drive that the files were on. In my case:
umount /dev/mapper/vg_ogre-lv_root

2) Well, that's my root drive. So before rebooting to my Fedora 12 Live CD, I checked that Fedora had the extundelete program in its repository. I was in luck!
[root@localhost ~]# yum install extundelete
Loaded plugins: presto, refresh-packagekit
fedora/metalink

 20 kB 00:00
fedora  4.2 kB 00:00
fedora/primary_db  9.7 MB 00:10
updates/metalink  16 kB 00:00
updates  4.5 kB 00:00
updates/primary_db  4.3 MB 00:01
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package extundelete.i686 0:0.1.8-2.fc12 set to be updated
--> Finished Dependency Resolution

You can also compile from source at the link above.

3) Since extundelete requires you to access an unmounted filesystem, I rebooted to Fedora 12 Live CD

4) I installed extundelete from the Fedora Updates repository
Transaction Summary

Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 52 k
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
updates/prestodelta  4.4 kB 00:00
Processing delta metadata
Package(s) data still to download: 52 k
extundelete-0.1.8-2.fc12.i686.rpm  52 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : extundelete-0.1.8-2.fc12.i686 1/1

Installed:
extundelete.i686 0:0.1.8-2.fc12 
Complete!

5) In order to restore deleted files, you must have a partition mounted that has enough space for the recovered files. I have a second partition (/mnt/backups) that I use to backup my main root partition. So while running under the Live CD, I created a destination directory and mounted /mnt/backups:
[root@localhost /]# mkdir /mnt/backup
[root@localhost /]# mount -t ext3 /dev/mapper/vg_ogre-lv_backup /mnt/backup

6) Though you can specify that extundelete undeletes individual files or all files on a filesystem, I ran extundelete with the proper command switches to undelete an entire directory of files:
[root@localhost backup]# extundelete /dev/mapper/vg_ogre-lv_root --restore-directory "mnt/videos/stormpigs/20100408"

WARNING: Extended attributes are not restored.

Loading group metadata ... 27025 groups loaded.

Loading journal descriptors ... 30695 descriptors loaded.

Writing output to directory RECOVERED_FILES/

Searching for recoverable inodes in directory mnt/videos/stormpigs/20100408 ...
309 recoverable inodes found.

Looking through the directory structure for deleted files ...
Restored inode 2495842 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/20100408_2.m2t

Restored inode 2495844 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/20100408_3.m2t

Restored inode 57147521 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/vadd.txt

Restored inode 57147522 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/aadd.txt

Restored inode 57147525 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/temp.txt

Restored inode 57147528 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/test.xml

Restored inode 57147532 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/testLabels.sh

Success..or close enough to it
This is good..I've undeleted all but one of the files I had deleted. So, it looks like before I unmounted my root partition, the inode for "20100408_1.m2t" was overwritten. Oh well, two out of three ain't bad.

Update 2011/10/05
This works nicely with ext4 partitions as well.
*** end update ***

Anyway, this short post should give you some comfort that extundelete actually does what it is supposed to. Thanks number9652!!

Reference
http://extundelete.sourceforge.net/
http://sourceforge.net/mailarchive/forum.php?forum_name=extundelete-users

automating repetitive tasks by scripting Cinelerra, part II

2010-04-12T20:16:00.030-04:00

This is a follow up to my earlier article:
/2010/01/automating-repetitive-tasks-by.html

Saving Time
In this entry, I discuss writing a shell script that automatically adds a second video track to your Cinelerra project. This comes in handy for projects that you do repeatedly. For example, a weekly podcast that uses a standard watermark.

For instance, I produce a monthly podcast for my band. You can see the pig watermark in the screen cap below:

In Cinelerra, the pig watermark is created by adding a second video track to my base Cinelerra project. This second video track includes a jpg stretched to the length of the first video track. Also, there is a chroma key effect applied to the pig track in order for it to be transparent against the background of the jam video.

The two items, a graphic of a pig and a video track with the image of the pig and applied effects are represented by XML in the Cinelerra EDL. Remember, the Cinelerra EDL (edit decision list) is the XML file that represents all the edits that you've performed in creating your video masterpiece. It looks like this:

In this blog post, I will show you a shell script that inserts these two chunks of XML into a basic Cinelerra project file. For my script to work properly, note that the base project needs at least one video track and at least one label. I'll get into the reason for that later. The basic project file can also have audio tracks as well.

The Asset, the Asset EDL and the Video Track EDL
One asset and two sections of XML represent the video track of the pig:
1) Here is the graphic of the pig, as a Google Doc.

2) Here is the EDL of the asset (pig2.jpg):

The EDL is also available in a Google Doc.

3) Here is the EDL of the video track of the pig:

The full EDL is also available as a Google Doc.

Since I use the pig watermark on each podcast, I reuse the two chunks of XML above for each installment of my monthly podcast. So I've saved both the asset and the video track XML above from an older project to separate files (pigAsset.txt and pigVideoTrack.txt). Using the shell script, I will insert both of these files at the appropriate places in my base Cinelerra project (my latest podcast). To make life simple, my base project has only one track, the video track of the band rehearsal.

Disclaimer
You're probably saying that this seems like a lot of work and you maybe getting confused. Well..it kind of is a pain in the ass, but it will be worth it when you can whip out a shell script and automatically add your watermark to your project! But I digress..

Shell Script Fun
Here is the meat of the script:
PWD=/mnt/scripts/scriptsTimeline/addPig
ASSET=$PWD/pigAsset.txt
TRACK=$PWD/pigVideoTrack.txt
OLDLENGTH=85965
NEWLENGTH=$(videoLength.sh $INPUT)

echo "OLDLENGTH is $OLDLENGTH"
echo "NEWLENGTH is $NEWLENGTH"
echo "INPUT is $INPUT"
echo "OUTPUT is $OUTPUT"

# adds both via a pipe WORKING
sed -e "s/^<\/ASSETS>/<\!-- $REM -->/g" $INPUT | sed -e "/<\!-- $REM -->/r $ASSET" | sed -e "/LABEL TIME/,/^<\/LABELS/{ /<\/LABELS>/r $TRACK
}" | sed -e "s/$OLDLENGTH/$NEWLENGTH/g" > $OUTPUT
The full script is here, again as a Google Doc.

The core of the script revolves around a neat bit of sed (stream editor) trickery that I mentioned in my first scripting article. In this case, the core function of the script does something similar, but still hinges upon the appearance of labels in the EDL ("LABEL TIME") as a positional reference within the EDL to perform the insertion:

sed -e "s/^<\/ASSETS>/<\!-- $REM -->/g" $INPUT | sed -e "/<\!-- $REM -->/r $ASSET" | sed -e "/LABEL TIME/,/^<\/LABELS/{ /<\/LABELS>/r $TRACK
}" | sed -e "s/$OLDLENGTH/$NEWLENGTH/g"

If we break this command apart, we see that it does a few things:
1) it adds the pig as an asset (the $ASSET constant in the second sed command) to a project (the $INPUT constant listed in the first sed command)

sed -e "s/^<\/ASSETS>/<\!-- $REM -->/g" $INPUT | sed -e "/<\!-- $REM -->/r $ASSET"

2) it adds a video track of said pig (the $TRACK constant) with this part of the command

| sed -e "/LABEL TIME/,/^<\/LABELS/{ /<\/LABELS>/r $TRACK

3) to make the new track the same length as the new track, it calculates the length of the base track and then substitutes that value for the initial length of the pig track.

| sed -e "s/$OLDLENGTH/$NEWLENGTH/g"

Now, I threw a curveball in there with that NEWLENGTH constant. This number is the current length of the video track in your input file and it is calculated via a second, slightly shorter script called "videoLength.sh".

Here is the Google Doc of that script.

Putting It All Together
To test how the script adds a track, download the following components to the same directory:
1) the jpg of the pig, save as pig2.jpg
2) the xml representing the pig, save as pigAsset.txt
3) the xml representing a new video track with effects, save as pigVideoTrack.txt
4) a test Cinelerra project, save as testProject.xml
5) the script, save as addPig.sh
6) the videoLength shell script

Don't forget to make both scripts executable:
chmod a+x addPig.sh
chmod a+x videoLength.sh, save as videoLength.sh

Script usage:
./addPig.sh [inputEDLfile] [outputEDLfile]

ex.
./addPig.sh testProject.xml trackAdded.xml

You may want to load testProject.xml into Cinelerra first to see that it only has one track. The script will create a new EDL file once it is run. Once you run the script, load your output EDL into Cinelerra. You should see the new track with the associated effects. The track should be the same length as testProject's video track.

Most likely, though, you won't see the pig, just because the pig is not in the same asset directory as the EDL specifies. Simply change pigAsset.txt to reference the pig into the proper directory, rerun the script and all should be good!

Summary
These scripts definitely can be improved upon, especially by someone who has better tools than sed to manipulate XML. But it was fun to automate a mundane task in Cinelerra. Now I don't have to manually add that watermark to my project anymore and neither do you!

the mule

timelapse in Cinelerra

2010-03-27T15:29:00.008-04:00

I've been playing around for the past few weeks with DSLR Remote, an iPhone app that allows you to control recent Canon or Nikon cameras remotely. DSLR Remote has an intervalometer, or timer, to allow you to take N number of photographs at T intervals. Once you take a bunch of photos, you can assemble them in Cinelerra to create a video of the change seen over time in the pictures, or timelapse.

Overview
This is my first experimentation with timelapse; specifically, timelapses of clouds and nighttime city scenes. I'm using a Canon 5D Mark II. Timelapse has been a real challenge, as the camera must be setup for proper exposure. I'm generally satisfied with the result, but I think it can be improved upon once I figure out the proper exposure settings.

An Education
I have learned some basics over the eight or so timelapses I've done:
1) don't use Automatic mode or full manual because the exposures vary quite a bit if the aperature (F Stop) or shutter speed change dramatically or conversely, don't change. Use aperature priority mode instead. Also, if the camera is in Auto mode, sometimes it will take time to change its settings. This will change the time interval between shots and make your final timelapse production jump around and jerky.

From the EXIF data in the JPEGs that I took during a morning timelapse, you can see how my shutter speed bounces around as the sun rose over a period of about an hour (FStop and ISO were locked):
[mule@ogre cinelerra]$ ./imageIdentifyFIsoShutter.sh /mnt/videos/projects/timelapse/Remote04639.jpg
exif:ExposureTime: 1/5
exif:FNumber: 11/1
exif:ISOSpeedRatings: 640
[mule@ogre cinelerra]$ ./imageIdentifyFIsoShutter.sh /mnt/videos/projects/timelapse/Remote05639.jpg
exif:ExposureTime: 1/100
exif:FNumber: 11/1
exif:ISOSpeedRatings: 640
[mule@ogre cinelerra]$ ./imageIdentifyFIsoShutter.sh /mnt/videos/projects/timelapse/Remote06239.jpg
exif:ExposureTime: 1/500
exif:FNumber: 11/1
exif:ISOSpeedRatings: 640

This can also blowout highlights and generally wash out the colors in the image, as shown in the latter half of this video:

2) shoot in aperature priority mode with auto ISO on. Auto ISO will account for variations in lighting conditions, like when a cloud passes in front of the sun.

3) make sure any lens stabilization is turned off, as this will make the frame jump around at times. This sounds counterintuitive, yes, but if the camera is on a tripod (as it should be for timelapse photography) a lens stabilizer will try to correct for movement that isn't present and will shift the frame around slightly. I figured this out after about five tries. (Yes, I am a doofus).

4) turn off automatic white balancing as it will effect the exposure from frame to frame

5) if you want to emphasize depth of field during day timelapses, make sure to get a neutral density filter. This will allow you to keep your aperature open wider, thus creating greater depth of field.

6) slower shutter speeds slightly blur movement. For example, daytime time lapse photography of clouds seem smoother with a slower shutter speed. Again, try a neutral density filter to help slow down the shutter speed. Of course, slow shutter speeds are very cool for time lapse traffic shots.

7) make sure your intervalometer is functioning properly. If it doesn't accurately time the snapshots, your final timelapse will look jerky and jump around once stitched together in Cinelerra.

Proper Interval
My camera has a limitation of 999 total shots that can be taken. In my experiments, I found that to acheive a smooth movement of clouds in my timelapses, I should take a picture every two or three seconds. That means if I want to take a picture every two seconds (30 pictures per minute) and I have 999 total shots available, my timelapse will last for a period of 33.3 minutes (999 shots / 30 pictures/minute). Doesn't seem that long, does it? If I chose an interval between shots of three seconds, my timelapse would occur over 49.95 minutes (999 shots / 20 pictures/minute). That seems a little more useful.

Timelapse in Cinelerra
Once I have my 999 images, I then stitch them together in Cinelerra at 60fps, 30fps or 15fps depending on my desired result..fast, medium or slower motion of whatever I was time lapsing. You'll need to do a bit of calculation to figure out at what duration (length) the images will be once imported into Cinelerra. That duration is set in Preferences -> Recording (screen below). I've provided those calculations below:
15 seconds = .0667
30 seconds = .0333
60 seconds = .0167

Be aware that if you import that many JPEGs, and especially if the JPEGs are very large, the import can take a while. My source files were 5616x3744 and took about 15-20 minutes to load.

Here's my acceptable timelapse..not the best..but getting there:

Going forward, the one thing I'd like to acheive is to remove or limit the slight changes in exposure that occur every once in a while so that the timelapse flows smoothly from one frame to another.

another work in progress..
the mule

Canon 5D: Best Modes for Video and Photos

fsarchiver, good backup for ext4 partitions

2010-02-21T08:26:00.012-05:00

Since I had lost my root partition the other day (!), I needed a decent method to backup my new ext4 partitions. Since partimage does not currently support ext4, I found fsarchiver:
http://www.fsarchiver.org/

Overview
I've partitioned my new 4.5TB drives this way:
/boot
/root 3.7TB
/backup 800GB

I formatted the backup filesystem as ext3. This way, I can simply boot with a Fedora Live CD, mount the backup partition and backup my boot and root partitions to the backup. Of course, I'll need to roll that backup off to another storage media. But this strategy helps when I make major updates to my system because I can easily rollback to an earlier version that is stored locally.

Most importantly, restore works!

Detail
Here's what I did the other day to get 'er going.

First, I booted to my Fedora Live CD. It didn't have fsarchiver installed by default, so I did so. You need to become superuser to do this:
[liveuser@localhost ~]$ su
[root@localhost liveuser]# yum install fsarchiver
Loaded plugins: presto, refresh-packagekit
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package fsarchiver.i686 0:0.6.7-1.fc12 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================
Installing:
fsarchiver i686 0.6.7-1.fc12 updates 93 k

Transaction Summary
=============================================================================================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 93 k
Is this ok [y/N]: y
Downloading Packages:
Transaction Test Succeeded
Installing : fsarchiver-0.6.7-1.fc12.i686 1/1
Installed:
fsarchiver.i686 0:0.6.7-1.fc12

Complete!

I created my backup directory and mounted it:
[root@localhost liveuser]# mkdir /mnt/backup
[root@localhost liveuser]# mount -t ext3 /dev/mapper/vg_ogre-lv_backup /mnt/backup
[root@localhost liveuser]# ls /mnt/backup
lost+found test.txt

Finally, I ran fsarchiver to do the backup and took advantage of its multithreaded capability:
[root@localhost liveuser]# fsarchiver -j7 -o savefs /mnt/backup/lv_root_backup.fsa /dev/mapper/vg_ogre-lv_root
Statistics for filesystem 0
* files successfully processed:....regfiles=306990, directories=31024, symlinks=16561, hardlinks=4157, specials=28
* files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0

You could concatenate these steps into a script:
[mule@ogre ~]$ cat systemBackup.sh
su -
yum install fsarchiver
mkdir /mnt/backup
mount -t ext3 /dev/mapper/vg-ogre/lv_backup /mnt/backup
fsarchiver -j7 -o savefs /mnt/backup/lv_root_backup.fsa /dev/mapper/vg_ogre-lv_root

Voila! In seven hours, I backed up approximately 450GB of data:
[root@localhost liveuser]$ ll /mnt/backup
total 459289180
drwx------. 2 root root 16384 2010-02-10 20:09 lost+found
-rw-r--r--. 1 root root 480252970629 2010-07-21 07:34 lv_root_backup.fsa

Update:
When my RAID set was not being checked, an fsarchive of about 760GB took 3.5 hours. Not bad!
***end update***

Restore
Restore works in a similar way. Since fsarchiver allows you to backup multiple filesystems within one archive, you need to specify which filesystem is getting restored:
fsarchiver restfs /mnt/backup/lv_root_backup.fsa id=0,dest=/dev/mapper/vg_ogre-lv_root

Nice.
the mule

References
http://www.fsarchiver.org/QuickStart

compiling Cinelerra on Fedora 12, x86-64 from source

2010-02-11T20:47:00.010-05:00

Happily, I'm over the hurdle with my server rebuild from Fedora 10 to Fedora 12 and have successfully built Cinelerra from source on my Fedora 12, x86-64 box! Yes!

The install was generally smooth, with the exception of a minor glitch: as aac encoding is proprietary, ffmpeg does not include faac support. Doh! So, you'll need to compile and install ffmpeg with faac support from source as per these excellent instructions from Doran:
http://fozzolog.fozzilinymoo.org/tech/2009/11/recompiling-ffmpeg-for-fedora-12-to-add-faac-support.html

RPM Fusion Yo!
Note that you'll need both the free and non-free repositories from RPM Fusion enabled in yum. Don't use the ATrpms repos if you're going to compile from source. I've had strange system problems when I've used them in combination with the Fusion repos.

So here's the deal..

Summary Steps to Installing Cinelerra from Source

download source code
install dependencies for ffmpeg compile
compile ffmpeg with faac support for fedora 12
install rest of Cinelerra dependencies
compile cinelerra

Detailed Steps to Installing Cinelerra from Source
1. Get the Cinelerra source code
git clone git://git.cinelerra.org/j6t/cinelerra.git my_cinelerra

2. Install dependencies for ffmpeg compile
Feel free to use the following script (don't forget to chmod a+x it!):
yum install dirac-devel faac-devel faad2-libs gsm-* imlib2-devel lame-libs* libdc1394-devel libvdpau-devel openjpeg-devel schroedinger-devel speex-devel texi2html x264-* xvidcore-* yasm

3. Compile ffmpeg with faac support

Highlights
[sodo@computer ~]$ yumdownloader --source ffmpeg

[sodo@computer ~]$ rpm -ivh ffmpeg-0.7.6-1.fc15.src.rpm

[sodo@computer ~]$ vi rpmbuild/SPECS/ffmpeg.spec
[sodo@computer ~]$ rpmbuild -ba ~/rpmbuild/SPECS/ffmpeg.spec --with faac

[sodo@computer ~]$ cd rpmbuild/RPMS/
[sodo@computer RPMS]$ cd x86_64/
[sodo@computer x86_64]$ ls -ltr
total 39164
-rw-rw-r--. 1 sodo sodo 259049 Dec 18 14:56 ffmpeg-0.7.6-1.fc15_fozz.x86_64.rpm
-rw-rw-r--. 1 sodo sodo 3138877 Dec 18 14:56 ffmpeg-libs-0.7.6-1.fc15_fozz.x86_64.rpm
-rw-rw-r--. 1 sodo sodo 166417 Dec 18 14:56 ffmpeg-devel-0.7.6-1.fc15_fozz.x86_64.rpm
-rw-rw-r--. 1 sodo sodo 9860213 Dec 18 14:56 ffmpeg-debuginfo-0.7.6-1.fc15_fozz.x86_64.rpm
[sodo@computer x86_64]$ sudo rpm -Uvh ffmpeg-libs-0.7.6-1.fc15_fozz.x86_64.rpm ffmpeg-0.7.6-1.fc15_fozz.x86_64.rpm ffmpeg-devel-0.7.6-1.fc15_fozz.x86_64.rpm ffmpeg-debuginfo-0.7.6-1.fc15_fozz.x86_64.rpm
[sudo] password for sodo:
Preparing... ########################################### [100%]
1:ffmpeg-libs ########################################### [ 25%]
2:ffmpeg ########################################### [ 50%]
3:ffmpeg-devel ########################################### [ 75%]
4:ffmpeg-debuginfo ########################################### [100%]

4. Install rest of Cinelerra dependencies
Feel free to use the following script (don't forget to chmod a+x it!):
yum install gsm-devel libvorbis* libogg* libtool* libtheora* libpng* libjpeg* libtiff* esound* audiofile* libraw1394* libavc1394* freetype* fontconfig* nasm e2fsprogs* OpenEXR* fftw fftw-devel libsndfile* libiec61883* libdv* libquicktime ffmpeg xvidcore* lame lame-devel a52* faad2* x264* mjpegtools* faac* vlc*

5. Compile Cinelerra
If you used the same default directory when you downloaded the source code, cd into that directory and type:
./autogen.sh

This will build your configuration files. Next, to configure Cinelerra with default options, type:
./configure

If you want to customize your configuration, just type "./configure --help". At which point, configure will spill its guts to you about all the Secrets of Cinelerra.

Then, to compile the code, run:
make

If you have multiple processors, you can even run "make -j N" where "N" is the number of processors you have minus one.

Finally, if make was successful or you are just feeling good about yourself, go ahead and install Cinelerra:
make install

Hopefully, this post has gotten you started on your way to using Cinelerra, Linux's "50,000 watt flamethrower of multimedia of editing"

Much thanks to Jack Crossfire, our patron saint of compositing code.

the mule

Reference
Don't forget to read the manual!

lost root partition..oops.

2010-02-06T18:29:00.015-05:00

I was running some disk performance statistics on the new Fedora 12 64-bit yesterday according to the very good benchmarking article on 3ware's site:

Benchmarking the 3ware 9000 Controller Using Linux Kernel 2.6

Benchmarking

I was benchmarking the write performance of my RAID set when it seemed to stall out. The process I was running was writing a bunch of zeros to a 20 gigabyte file. I believe the stall was due to the fact that my RAID controller card's battery was disconnected; hence, write-cacheing was disabled.

I let the process try to finish for four hours. I figured it should have finished writing that 20GB file by that time. However, the fact that the system was still slow to non-responsive indicated that activity was still taking place. But, being an impetuous fool, I was anxious to get working on some video and also thought it might be an interesting test of the resilience of the ext4 filesystem if I just shut the system down. So I as a soft reboot did not do the trick, I hard powered the box off.

Sleeping the Sleep of the Dead

In retrospect, I should have let the box finish whatever it was doing, because as you may have guessed it, the box didn't come back up. Here was the first indication from the kernel messages:

Boot has failed..sleeping forever

And in the dmesg output:

can't mount root filesystem

can't access tty job control turned off

Woops. Dracut did find my volume group:

dracut: 2 logical voumes in "vg_ogre" now active

Something was wrong with the root filesystem mount:

mount: you must specify the filesystem type

Just in case, I rebooted with the following kernel parameters in grub to see more debugging and to drop me to an emergency shell to see if I could debug the problem:

kernel .. debug rdshell

What Up, ext4?

Oh boy. So, ext4 is not as resilient as I believed. I thought the best course of action would be to load up Fedora Live, and look at the disk stats. Since fdisk does not work with GPT partitions, I used parted and thought that I'd use e2fsck to fix any bad blocks. After booting the Live CD, here's what I found:

The swap drive seemed in tact (oh, great):

[liveuser@localhost ~]$ dmesg | grep vg
vgaarb: device added: PCI:0000:07:00.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
Adding 12369912k swap on /dev/mapper/vg_ogre-lv_swap. Priority:-1 extents:1 across:12369912k

I thought I'd try to manually mount my / partition. I had to become superuser in order to do this:

[liveuser@localhost ~]$ su
[root@localhost liveuser]# mkdir /mnt/root
[root@localhost liveuser]# mount -t ext4 /dev/mapper/vg_ogre-lv_root /mnt/root
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_ogre-lv_root,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

Dmesg tells me what I already know:
[root@localhost liveuser]# dmesg | tail
[drm] nouveau 0000:07:00.0: 0x00409910: 0x3fbf3fdb
[drm] nouveau 0000:07:00.0: 0x00409e08: 0x0002dea8
[drm] nouveau 0000:07:00.0: 0x00409e0c: 0x00000000
[drm] nouveau 0000:07:00.0: 0x00409e24: 0x0a21026f
EXT4-fs (dm-2): VFS: Can't find ext4 filesystem

I just want to see what fdisk reads about my hardware RAID5 array (3ware 9650SE):

[root@localhost liveuser]# fdisk -l /dev/sda

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.5 TB (4499967049728 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID
partition table format (GPT).

Disk /dev/sda: 4500.0 GB, 4499967049728 bytes
255 heads, 63 sectors/track, 547089 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f0844

Device Boot Start End Blocks Id System
/dev/sda1 1 267350 2147483647+ ee GPT

What does parted see about /dev/sda?

[root@localhost liveuser]# parted /dev/sda print
Model: AMCC 9650SE-4LP DISK (scsi)
Disk /dev/sda: 4500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 17.9kB 210MB 210MB ext4 boot
2 210MB 4500GB 4500GB lvm

At least the partition is there. But it looks like parted does not have support for checking ext4 filesystems yet:
[root@localhost liveuser]# parted /dev/sda
GNU Parted 1.9.0
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) check 1
No Implementation: Support for opening ext4 file systems is not implemented yet.
(parted) check 2
Error: Could not detect file system.
(parted) quit

e2fsck bound!

Let me run e2fsck (which does have support for ext4 filesystems) and see if I can fix the problem:

[root@localhost liveuser]# e2fsck
Usage: e2fsck [-panyrcdfvtDFV] [-b superblock] [-B blocksize]
[-I inode_buffer_blocks] [-P process_inode_size]
[-l|-L bad_blocks_file] [-C fd] [-j external_journal]
[-E extended-options] device

Emergency help:
-p Automatic repair (no questions)
-n Make no changes to the filesystem
-y Assume "yes" to all questions
-c Check for bad blocks and add them to the badblock list
-f Force checking even if filesystem is marked clean
-v Be verbose
-b superblock Use alternative superblock
-B blocksize Force blocksize when looking for superblock
-j external_journal Set location of the external journal
-l bad_blocks_file Add to badblocks list
-L bad_blocks_file Set badblocks list

My skills at e2fsck are pretty basic. I use the -n option to make no changes while I review what e2fsck finds out about the array:

[root@localhost liveuser]# e2fsck -n /dev/mapper/vg_ogre-lv_root
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Superblock invalid, trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear? no

e2fsck: Illegal inode number while checking ext3 journal for /dev/mapper/vg_ogre-lv_root

Invalid journal..oops.

[root@localhost liveuser]# e2fsck -v /dev/mapper/vg_ogre-lv_root
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Superblock invalid, trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear?y

I had thought that ext4 gave us the safety of a journalled filesystem (like ext3) with increased performance. You would have thought it could have recovered from being shutdown while writing a bunch of zeros to a 20 gigabyte file.

And then of course, hundreds to thousands of these various errors:

Group descriptor 32923 checksum is invalid. FIXED.

Entry 'e61abf8156cc476151baa07d67337cae-le64.cache-3' in ??? (57347) has deleted/unused inode 212. Clear? yes

Unconnected directory inode 98305 (...)
Connect to /lost+found? yes

Free blocks count wrong for group #138 (32768, counted=557).
Fix? yes

Free inodes count wrong for group #308 (8192, counted=8186).
Fix? yes

Directories count wrong for group #308 (0, counted=6).
Fix? yes

Finally..at the bottom of the list of errors:
Recreate journal? yes

Creating journal (32768 blocks): yyyyyyy Done.

*** journal has been re-created - filesystem is now ext3 again ***

/dev/mapper/vg_ogre-lv_root: ***** FILE SYSTEM WAS MODIFIED *****

327475 inodes used (0.12%)
585 non-contiguous files (0.2%)
130 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 310327/414/1
239381919 blocks used (21.85%)
0 bad blocks
42 large files

283167 regular files
27385 directories
0 character device files
0 block device files
0 fifos
3953 links
16849 symbolic links (16659 fast symbolic links)
63 sockets
--------
331417 files
[root@localhost liveuser

Result?
So let's see if I have files in tact after that 18 hour experience..
[root@localhost liveuser]# mount -t ext4 /dev/mapper/vg_ogre-lv_root /mnt/root/
[root@localhost liveuser]# ls /mnt/root
lost+found
[root@localhost liveuser]# ls /mnt/root
lost+found
[root@localhost liveuser]# ls /mnt/root/lost+found/
*348489 *723483 324843 238390

Ah..that would be a "no." Time to reinstall F12. Ugh. Lesson learned. But I need to know why I couldn't recover a journal. Maybe I did not look in the right place. I need to understand journalling better.

Things I Learned Along the Way

Some boot info from the Live CD

[root@localhost liveuser]# grep EFI_ /boot/config-2.6.31.5-127.fc12.i686
CONFIG_EFI_VARS=y
CONFIG_EFI_PARTITION=y

You can force a filesystem check upon the next reboot with this command:

shutdown -rF now

You can run verbose debug messages and drop to an emergency shell by placing these commands on the kernel line in grub:

kernel /vmlinuz-2.6.31.12-174.2.3.fc12.x86_64 ro root=/dev/mapper/vg_ogre-lv_root debug rdshell

References

Ext4 Howto - Ext4
Recover lost partition table on GPT using testdisk

TestDisk - Undelete file for ext2

Linux Creating a Partition Size Larger than 2TB

Fedora 12 common bugs

time average video effect

2010-02-04T20:13:00.009-05:00

Time average is a neat effect. It takes a number of frames of video to average or accumulate.

:averaging
In the below video, I show averaging. Averaging lots of frames gives a very ghost-like effect. Average very few frames shows trails of whatever movement is happening. Medium settings are like you've had too much to drink.

Cinelerra time average effect from crazed mule on Vimeo

:accumulate
A second option called Accumulation is just that..stacking one frame on top of the other. If the scene in the frame is well lit, the frame will blowout..make it so bright that all detail will be lost. The detail gets lost quickly too, when just a few frames are being accumulated. Accumulate is more useful for extremely dark scenes, like making very distant stars become more visible, as in the example below. In the example, I show the original and the time averaged result:

Cinelerra time average effect: accumulate from crazed mule on Vimeo

There is another time averaging parameter called "Inclusion Or" that I have not used. It is explained in the link to the manual below as are the two options "Reprocess Frame Again" and "Disable Subtraction."

cheers,
the mule

Reference
Cinelerra User Manual

mpeg toc maker

2010-01-30T12:14:00.005-05:00

I got tired of having Cinelerra hang while create table of contents files for a list MPEG videos. So I wrote my own script to create the table of contents files for a list of MPEGs in a directory. Once you specify the LIST, the script plops the table of contents files in your users' .bcast directory in ~/.bcast.

I output the file name, start time, end time and elapsed time for the table of contents creation. This output looks like this:


SCRIPTSTART:                  20100130_121201
       FILE:                     mvi_0703.m2t      START: 20100130_121201        END: 20100130_121204    TIME(s):       3
       FILE:                     mvi_0705.m2t      START: 20100130_121204        END: 20100130_121222    TIME(s):      18
       FILE:                     mvi_0706.m2t      START: 20100130_121222        END: 20100130_121255    TIME(s):      33
       FILE:                     mvi_0708.m2t      START: 20100130_121255        END: 20100130_121325    TIME(s):      30
       FILE:                     mvi_0709.m2t      START: 20100130_121325        END: 20100130_121341    TIME(s):      16
       FILE:                     mvi_0710.m2t      START: 20100130_121341        END: 20100130_121347    TIME(s):       6
       FILE:                     mvi_0711.m2t      START: 20100130_121347        END: 20100130_121401    TIME(s):      14
       FILE:                     mvi_0713.m2t      START: 20100130_121401        END: 20100130_121403    TIME(s):       2
       FILE:                     mvi_0714.m2t      START: 20100130_121403        END: 20100130_121415    TIME(s):      12
       FILE:                     mvi_0715.m2t      START: 20100130_121415        END: 20100130_121415    TIME(s):       0
       FILE:                     mvi_0716.m2t      START: 20100130_121416        END: 20100130_121416    TIME(s):       0
       FILE:                     mvi_0717.m2t      START: 20100130_121416        END: 20100130_121421    TIME(s):       5
       FILE:                     mvi_0718.m2t      START: 20100130_121421        END: 20100130_121442    TIME(s):      21
       FILE:                     mvi_0719.m2t      START: 20100130_121442        END: 20100130_121442    TIME(s):       0
       FILE:                     mvi_0720.m2t      START: 20100130_121442        END: 20100130_121446    TIME(s):       4
       FILE:                     mvi_0721.m2t      START: 20100130_121446        END: 20100130_121449    TIME(s):       3
       FILE:                     mvi_0722.m2t      START: 20100130_121449        END: 20100130_121454    TIME(s):       5
       FILE:                     mvi_0724.m2t      START: 20100130_121454        END: 20100130_121455    TIME(s):       1
       FILE:                     mvi_0725.m2t      START: 20100130_121455        END: 20100130_121507    TIME(s):      12
       FILE:                     mvi_0727.m2t      START: 20100130_121507        END: 20100130_121515    TIME(s):       8
       FILE:                     mvi_0728.m2t      START: 20100130_121515        END: 20100130_121531    TIME(s):      16
       FILE:                     mvi_0730.m2t      START: 20100130_121531        END: 20100130_121534    TIME(s):       3
       FILE:                     mvi_0731.m2t      START: 20100130_121534        END: 20100130_121535    TIME(s):       1
  SCRIPTEND:                  20100130_121535 RUNTIME(s):             213

*** Update 2010/05/28 ***
I added a little logic to the beginning of the script that checks for the presence of a single argument to the script. That argument is the name of a single file to convert. So now, you have two choices:
1) convert a list of mpeg files as specified in the LIST variable or
2) convert a single file via the command line like so:
./tocMaker.sh videoToConvert.m2t
*** end update ***

Here is the script:


#!/bin/bash
#
# tocMaker.sh
# Author: SCF
# Website: http://crazedmuleproductions.blogspot.com
#
# Purpose: This script builds the table of contents files for a list of mpeg files
#
# Input: List of mpeg files as specified in the LIST variable or a single file via command line like so:
#  ./tocMaker.sh videoToConvert.m2t
#
# Output: Runs mpeg3toc and puts toc's in your users' .bcast directory
#
DEBUG=
if [ $1 ]
then
 LIST=$1
else
 LIST="*.m2t"
fi
echo "converting $LIST"
TOTALTIME=0
function dt {
       DATE=$(date +"%Y%m%d_%H%M%S")
       echo $DATE
}
function st {
       SECONDS=$(date +%s)
       echo $SECONDS
}
PWD=$(pwd)
DIR=$(echo $PWD | sed 's/\//_/g' | sed 's/^_//g')

SCRIPTSTART=$(dt)
printf "%13s %32s\n" "SCRIPTSTART:" $SCRIPTSTART

for FILE in $(ls -1 $LIST)
do
       START=$(dt)
       COUNTBEGIN=$(st)
       NEWFILE=$(echo $FILE | sed 's/\./_/g')
       FULLPATH="${DIR}_${NEWFILE}.toc"
       ORIG_PATH="$PWD/$FILE"
       if [ $DEBUG ]
       then
               echo "$FILE was: $FILE, is: $NEWFILE"
               echo "FULLPATH is $FULLPATH"
               echo "command is mpeg3toc $ORIG_PATH ~/.bcast/$FULLPATH"
       fi
       mpeg3toc $ORIG_PATH ~/.bcast/$FULLPATH
       END=$(dt)
       COUNTFINISH=$(st)
       ELAPSED=$(expr $COUNTFINISH - $COUNTBEGIN)
       printf "%13s %32s %11s %15s %11s %15s %11s %7d\n" "FILE:" $FILE "START:" $START "END:" $END "TIME(s):" $ELAPSED
       TOTALTIME=$(expr $ELAPSED + $TOTALTIME)
done
SCRIPTEND=$(dt)
printf "%13s %32s %11s %15d\n" "SCRIPTEND:" $SCRIPTEND "RUNTIME(s):" $TOTALTIME

cheers,
the mule

batch render redux

2010-01-27T01:26:00.016-05:00

As a follow up to this original post, I thought I'd give a few hints on working with the batch render function in Cinelerra. I've been using the batch render to prepare me for my new Fedora 12, x86-64 system. Specifically, I am using batch render to profile the capacity and speed of my current Fedora 10 system. After I install the new Fedora 12 system, I can then use the Fedora 10 performance baseline to tell me how much faster (or slower) the new system is in relation to the old system.

Selecting a Cinelerra Project
The first thing I did was use a short (~1 min) clip of a project that I had been working on:

I then used this project to output a short clip as the basis for the batch render:

Batch render is accessed by typing Shift-B within Cinelerra:

You can save a batch render list to XML format. This format is similar to the XML of the edit decision list (EDL) that Cinelerra stores when a project is saved. You can then load that XML to use later.

Batch Render Gotchas
I used a previous batch list to render out that short clip to about twenty different file formats. The batch render blew up a few times, so I had to get over a few obstacles:
1) the batch list I had saved months ago was out of date and the directory pointers in the XML were incorrect. I fixed the incorrect pointers by going into the vi text editor and doing a wholesale conversion with a sed construct:
:1,$ s/videos\/oldpath/videos\/newpath/g

1,$ says look for and make replacements on all lines. s means "search for.." The matching expression looks for the string "videos/oldpath" in the file and replaces it with "videos/newpath". g means "do the replacement globally"

2) my project had an improperly formatted Quicktime video track on the timeline. I kept getting Quicktime errors when the batch ran, which would crash Cinelerra. Once I removed that errant track, the batch render worked correctly.

3) once I got the batch running, my disk would fill up quickly as I was rendering to a few uncompressed formats. I deleted some very large, extraneous files and the batch was able to complete.

So you can see that you need to prepare both the base project and your system if you expect your batch render to run properly.

Command Line Batching
Once you get your batch working, another nice feature of batch rendering you can take advantage of is the ability of Cinelerra to do this at a command line. I just kicked off a batch job at the command line and see that it does work on a box with X installed. Nicely, it also give you an ETA:

[sodo@tbear ~]$ cinelerra -r /mnt/videos/cinelerra/batch/batchList.xml
/mnt/videos/cinelerra/batch/renderCompatibility.xml
Cinelerra 2.1CV (C) 2006 Heroine Virtual Ltd.
Compiled on Sat Jan 23 01:32:17 EST 2010

Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.
Render::run: /mnt/videos/cinelerra/batch/renderCompatibility.xml
Render::render: starting render farm
[mpeg4 @ 0x7ff313d3ba00]warning: first frame is no keyframe
[mpeg4 @ 0x7ff313d3ba00]concealing 1 DC, 1 AC, 1 MV errors
12% ETA: 0:07:45

Batch Results
So my goal was to profile my current system's capacity by capturing the system's cpu and disk utilization while the files rendered. To really see what is going on though, I thought it might be nice to have a graphical representation of the render as it occurs over time. So I spent some time writing a gnuplot script to plot the system utilization as the files rendered:

A bit of explanation may be required. I captured the output of vmstat to a file. VMstat has CPU load and wait i/o (disk utilization) statistics. While capturing that output, I kicked off the batch render.

In the graphic, you can see that different types of renders have different utilization profiles. For example, the mpeg4 renders were generally lower in CPU utilization (red line), while h264 renders used a lot of CPU. Similarly, the uncompressed formats like rgb/rgba/yuv420planar stress out the disk quite a bit (green line). Please excuse the fact that the filenames aren't perfectly lined up with each file's render profile..this was my first effort at graphing render times.

It will be interesting to see how the new Fedora 12 install affects the CPU. Also, I am planning on installing a new hardware RAID set, so I expect those green lines to go to zero (hopefully)!

ciao,
the mule

compile time performance improved!

2010-01-24T15:24:00.008-05:00

I was compiling Cinelerra today and noticed that CPU usage was very low during the compile..around 10-15% utilized. I have a dual CPU, quad core box. This makes for a total of eight processors. So with all those CPUs, I figured there must be a way to make compiling faster.

Actually, this low CPU use during compiles was something I had noticed the first time I installed Cinelerra. Ashamedly, I've forgotten to investigate this issue in the two years that I've had the box. So today I googled for "make compiler see multiple CPUs" and found the -j switch to "make" the program that does the compiling:
http://blogs.koolwal.net/2009/04/20/tip-compile-your-programs-fasters-with-multiple-processor-machines/

This article also mentions the CONCURRENCY_LEVEL environment variable, but that variable did not work for my box, a Dell SC1430. So I used the -j switch to make instead:
[mule@ogre my_cinelerra]$ make -j7
(CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /home/sfrase/my_cinelerra/missing --run autoheader)
rm -f stamp-h1

Here are my results.

the time it took to compile Cinelerra normally, without -j:
3min 45s
the time it took to compile Cinelerra with the -j8 (for eight cores):
1min 16s

Holy crap! That's a 300% speed improvement!

Glad I finally researched this.
scott

ps - one other note: 7z is a multithreaded version of tar. On Fedora, use 7za
Installing : p7zip-4.65-2.fc12.x86_64
7za - A file archiver with highest compression ratio

SYNOPSIS
7za [adeltux] [-] [SWITCH]

Related Posts
http://crazedmuleproductions.blogspot.com/2007/10/multithreading-in-ffmpeg-and-mpstat.html

Fedora 12, x86-64 upgrade

2010-01-24T10:39:00.008-05:00

The time has come again..sytem upgrade. Ugh.

From Fedora 10 x86-64 to Fedora 12 x86-64
I say "ugh", but I truly am excited as Fedora 12 does have some nice performance improvements (ext4, kernel modesetting, faster boot, rpm) that they've packaged since the Fedora 10 system I'm working with now:
http://fedoraproject.org/wiki/Releases/11/FeatureList
http://fedoraproject.org/wiki/Releases/12/FeatureList

To be clear, I don't do upgrades. I will tar up my /home directory to USB, install the new OS from scratch and then blast my /home directory onto the clean new OS and RAID array.

Thinking Hard
I've spent quite a bit of time planning this upgrade. One of the big things I am doing is to profile the performance of my system before and after the OS and hardware upgrades. Of course, I won't be able to determine whether or not the performance gain is coming from the OS or the new RAID array, but at the end of the day, I simply want to be able to say "my system is now X% faster."

I will be looking at the performance of the system from the OS, Cinelerra and encoder perspectives.

Learning about Fedora 12
http://fedoraproject.org/wiki/Common_F12_bugs
http://www.scribd.com/doc/24513176/Fedora-12-Installation-Guide
Changes_in_Fedora_for_Desktop_Users

Hardware changes going in
New RAID configuration:
3WARE Pci-e 9650SE RAID card with Battery Backup
four Western Digital 1.5 TB Green SATA 32MB Cache Hard Drive

Virtual Machine Testbed
One of the things that has helped me in the process is using VMware Server to test out Fedora 12. I've caught a couple things right off the bat: as it is a proprietary format, FAAC is not installed with FFmpeg by default. I was able to resolve this through Doran's excellent post here:
http://fozzolog.fozzilinymoo.org/tech/2009/11/recompiling-ffmpeg-for-fedora-12-to-add-faac-support.html

Also, H264 encoder magic has changed a bit. Other than that, my output testing to various formats (MPEG-PS, HDV, DVD, iPod/iPhone) has worked very well.

General prep work
work out bugs with Fedora 12 virtual machine
clean up old F10 system
backup F10 system files via script
backup /home directory via tar to external drive

Installation steps
Install new F12, Developer's edition
Install RPMs via script
Build and install FFmpeg RPM with faac support from nonfree RPM Fusion repo via script
Install favorite programs
Install Cinelerra dependencies
Install Cinelerra

For those with strong constitutions, here's the full project plan:
http://spreadsheets.google.com/ccc?key=0AjSzE_zejuQZdFphck9aQUVBbzZVOWhyOC1CaVFVQmc&hl=en

I'm almost there..most of the planning is done. Now, to execute! I'll let you know how it goes.
The Mule

Reference
http://www.graphics-muse.org/wp/?p=501

best practices to alleviate audio sync problems in Cinelerra

2010-01-16T15:30:00.011-05:00

Audio synchronization problems are the bane of video editors everywhere. Here's what I do in Cinelerra to try to prevent them.

The first thing to understand is that playback in Cinelerra is a separate process from rendering (exporting) your project to a final format. So it is a good idea to calibrate the audio-video synchronization of your files when they are played back in the Cinelerra compositor as well as the a/v sync of rendered output. More specifically:
1) You need to calibrate the audio you see being played back in the compositor. Do this by setting Audio Offset in the Preferences -> Playback -> Audio Out so that your playback audio is in sync with the video. I generally use ALSA as my audio driver under Preferences -> Playback. (You might also want to try to use OSS or ESound to see if those audio drivers help your playback sync issues.) For reference, my offset is .1 seconds.

2) You'll need to calibrate the audio and video once it is rendered out of Cinelerra. This doesn't have to be the absolute final format of the file. Just an intermediate file format to determine that audio is not drifting and in sync. You'll do this by setting Nudge on your audio tracks so that your rendered audio and video are in sync. Here's an article on nudge:
http://crazedmuleproductions.blogspot.com/2007/05/nudge-avidemux2-and-reminder-about.html

Let me give you an example.

My projects usually run about an hour on the timeline. When I start a project, I will concatenate a bunch of clips or raw videos on the timeline. I will then render a short segment of video, about 15 seconds long, from about 45 minutes into the timeline. The segment I take should have key features like someone speaking or someone hitting a drum that I can use to determine audio synchronization. I then render the file to a format that works well in the various Linux media players:
-Quicktime for Linux using MPEG4 video compression and MP4 or twos complement (pcm) audio compression
-Quicktime for Linux container using JPEG video compression and MP4 or twos complement

The side benefit of using MPEG4 video compression is that it renders damn fast.

The good thing about these steps is that if you work with the same format of video over and over, you won't need to change #1, the Audio Offset.

Here are notes regarding specific source formats and output formats.

MPEG SOURCE FILES to MPEG RENDERED OUTPUT
After much study and pain, I've found that when working with MPEG-TS HDV files as my source (and MPEG formats in general), the best pipe to render from Cinelerra in order to alleviate sync headaches is mpeg2enc. I'm not five channel yet, so I just export my 48.1Khz stereo audio as MPEG Layer II audio MP3s @ 384kbps. I then use mplex to combine my audio and video streams into a program stream. Further on, I use VLC to convert that PS into MPEG-TS (HDV) and then use FFMPEG to convert the HDV into DVD and iPod/iPhone compatible formats.

QUICKTIME (QT) SOURCE FILES to QT RENDERED OUTPUT
If my source video format is Quicktime for Linux (like from a screen capture), then I have no sync problems rendering out to QT for Linux.

MPEG SOURCE FILES to QT RENDERED OUTPUT
If I render my MPEG-TS, HDV projects using Quicktime for Linux as an output format (MPEG4 or JPEG video w/twos complement or MP4 audio compression), then I will get sync problems in the rendered output. I then follow the above best practices to alleviate the problem.

TEST YOUR OUTPUT
In Fedora, I've found that the best media players to test final output are mplayer, vlc and ffmpeg.

Finally, there are options to mplex and ffmpeg to offset audio and video sync problems:
mplex -O ; eg, mplex -O 350
ffmpeg -itsoffset ; eg, ffmpeg -itsoffset 0.35

Hope this helps,
the mule

Reference
My Article on Cinelerra's "nudge" parameter
FFMPEG HowTo
http://man-wiki.net/index.php/1:mplex

automating repetitive tasks by scripting Cinelerra EDL, part I

2010-01-09T17:19:00.014-05:00

I produce a bi-monthly video of my band's jam sessions:
http://feeds.feedburner.com/StormpigsPodcast

The format of the video edit is roughly the same for every video:

intro titles
staff titles
songs
end credits

Since the Cinelerra EDL (edit decision list) is an XML text file, it occurred to me that I should be able to add the titles by editing this text file.

Cinelerra's Edit Decision List (EDL)
Let's look at Cinelerra's EDL. This is an XML text file that gets written when you save a Cinelerra project. The easiest way to review the EDL is to open it in a browser, as the browser will let you expand and collapse the XML elements. Here's a screen cap of the EDL in Firefox:

When I create my monthly video podcast, I place title effects on Cinelerra's timeline for each song of about a dozen songs. The title effect looks like this:

Placing title effects is a real time sink, especially when I have to place one for each song. I place title effects about 20 seconds into each song. I separate each song by placing a label between the clips. So the labels indicate where in the video track I will need to place title effects. You can see that the title effect is placed after the appearance of a label in Cinelerra's timeline below:

The Task
My goal is to have a shell script find the position of the label in the EDL and plop the title effects about 20s after the appearance of the label. After I add the labels programmatically, I can do fancier edits to the project later on.

In today's post, I will describe a bit more about my workflow and the EDL, and also show you a basic shell script command to insert a title effect into the EDL.

The Procedure
First, I put my basic edit together assembling the clips on the timeline, doing my audio and video fades and placing labels between the clips. This is my first round of edits. At this point, my timeline is very simple. I only have a video track and two audio tracks:

I will use a script to insert a title effect into the EDL file. In the Cinelerra EDL, a title effect (designated by the PLUGINSET XML element) looks like this:

Let's inspect the EDL in order to find the block of labels. If you grep for 'LABEL TIME' in the EDL, you'll find an XML code block that looks like this:

I will need to edit the video TRACK that appears directly beneath the LABELS in the EDL. Since there is quite a bit of confusing repetition in the EDL file, finding the list of LABELs helps identify which TRACK we need to edit. I've capitalized the words in the above sentence to highlight the XML elements you'll need to look for when editing EDL.

Using a bash shell script, I'll add a title effect after the last MASKAUTOS XML tag, but before the closing TRACK tag. Again, this is the closing TRACK tag of the video TRACK:

I've created a second file containing just one title effect, the PLUGINSET example from above:

I will then use a bit of sed (stream editor) magic to insert that file into the right spot in the main Cinelerra EDL for the project. Here is the script command that will place the PLUGINSET into the correct position in the XML:
sed -e "/LABEL TIME/,/^<\/MASKAUTOS/{ /<\/MASKAUTOS>/r titleEffectPluginset.txt
}" cinelerraEdl.xml > cinelerraEdlNewEdit.xml

Let's break this apart:
sed -e "/LABEL TIME/ # This finds the first occurrence of "LABEL TIME" (a label) in the EDL. Remember that the appearance of labels in the EDL tells us that the video TRACK in which we need to insert our title effect comes next in the file.

,/^<\/MASKAUTOS/{ # This finds the first occurrence of a closing MASKAUTOS tag after the string "LABEL TIME". Our title effect, the PLUGINSET, will be inserted after this closing tag.

/<\/MASKAUTOS>/r titleEffectPluginset.txt # Insert the title effect boilerplate after the end MASKAUTOS XML tag

}" cinelerraEdl.xml > cinelerraEdlNewEdit.xml # Have sed perform the edit on "cinelerraEdl.xml", but save the output to "cinelerraEdlNewEdit.xml

The Result - Before

The Result - After

You can see that sed has inserted the title effect (the PLUGINSET XML boilerplate) in between the closing MASKAUTOS tag and the closing tag of the video TRACK. Pretty cool!

This is a simple example of editing Cinelerra EDL, but is the first step to helping me automate otherwise manual tasks in my monthly video podcast creation.

I will try to expand upon this subject in future posts.

-enjoy-
The Mule

Cinelerra 4 just not working

2009-05-30T12:08:00.007-04:00

I was interested in trying out Cinelerra 4, Heroine Warrior's latest version of Cinelerra. To reiterate, there are two versions of Cinelerra available:
Heroine Warrior's, the original coded by Adam Williams (bow down to the man!) and
Cinelerra Community Version, Cinelerra CV

On my Fedora 10, x86-64 setup, I gave the latest HV Cinelerra 4 version a try. I encountered a bunch of hurdles, mainly the GCC 4.3 changes that caused missing header information in mjpegtools-1.9.0_rc3 listed here:
http://bugs.gentoo.org/show_bug.cgi?id=200767

I got the software to compile. At the final "make install" step, the installation starts, but does not complete with only the first 40 or so lines of the install finishing:
[mule@ogre bin]# make install
make -f build/Makefile.cinelerra install
make[1]: Entering directory `/usr/src/cinelerra-4'
make -C plugins install
make[2]: Entering directory `/usr/src/cinelerra-4/plugins'
mkdir -p ../bin/fonts
cp fonts/* ../bin/fonts
mkdir -p ../bin/shapes
cp shapes/* ../bin/shapes
cp ../thirdparty/mjpegtools*/mpeg2enc/mpeg2enc ../bin/mpeg2enc.plugin
make[2]: Leaving directory `/usr/src/cinelerra-4/plugins'
DST=../bin make -C libmpeg3 install
make[2]: Entering directory `/usr/src/cinelerra-4/libmpeg3'
cp x86_64/mpeg3dump x86_64/mpeg3peek x86_64/mpeg3toc
x86_64/mpeg3cat ../bin
make[2]: Leaving directory `/usr/src/cinelerra-4/libmpeg3'
make -C po install
make[2]: Entering directory `/usr/src/cinelerra-4/po'
mkdir -p ../bin/locale/de/LC_MESSAGES
cp sl.mo ../bin/locale/sl/LC_MESSAGES/cinelerra.mo
make[2]: Leaving directory `/usr/src/cinelerra-4/po'
make -C doc install
make[2]: Entering directory `/usr/src/cinelerra-4/doc'
mkdir -p ../bin/doc
cp arrow.png autokeyframe.png camera.png channel.png crop.png cut.png
expandpatch_checked.png eyedrop.png fitautos.png ibeam.png
left_justify.png magnify.png mask.png mutepatch_up.png paste.png
projector.png protect.png record.png recordpatch_up.png rewind.png
singleframe.png show_meters.png titlesafe.png toolwindow.png
top_justify.png wrench.png magnify.png ../bin/doc
cp: warning: source file `magnify.png' specified more than once
cp cinelerra.html ../bin/doc
make[2]: Leaving directory `/usr/src/cinelerra-4/doc'
cp COPYING README bin
make[1]: Leaving directory `/usr/src/cinelerra-4'
[mule@ogre bin]#

Therefore, the installation does not copy the cinelerra binary into /usr/local/bin. If I try to run the binary from the source code directory, I get this:
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/brightness.plugin:
undefined symbol: glUseProgram
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/deinterlace.plugin:
undefined symbol: glUseProgram
undefined symbol: glNormal3f
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/swapchannels.plugin:
undefined symbol: glUseProgram
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/threshold.plugin:
undefined symbol: glUseProgram
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/zoomblur.plugin:
undefined symbol: glEnd
signal_entry: got SIGSEGV my pid=3965 execution table size=16:
awindowgui.C: create_objects: 433
awindowgui.C: create_objects: 440
awindowgui.C: create_objects: 444
awindowgui.C: create_objects: 447
awindowgui.C: create_objects: 453
suv.C: get_cwindow_sizes: 744
suv.C: get_cwindow_sizes: 774
suv.C: get_cwindow_sizes: 800
suv.C: get_cwindow_sizes: 821
editpanel.C: create_buttons: 177
editpanel.C: create_buttons: 303
editpanel.C: create_buttons: 177
editpanel.C: create_buttons: 303
mwindowgui.C: create_objects: 192
mwindowgui.C: create_objects: 195
mwindowgui.C: create_objects: 199
signal_entry: lock table size=6
0x357dc80 RemoveThread::input_lock RemoveThread::run
0x64d0370 CWindowTool::input_lock CWindowTool::run
0x64f3640 TransportQue::output_lock PlaybackEngine::run
0x3441940 TransportQue::output_lock PlaybackEngine::run
0x3442420 MainIndexes::input_lock MainIndexes::run 1
0x3442f80 Cinelerra: Program MWindow::init_gui *
BC_Signals::dump_buffers: buffer table size=0
BC_Signals::delete_temps: deleting 0 temp files
SigHandler::signal_handler total files=0

Even though I have an NVidia graphics card, the error lines above were related to OpenGL. Thus, I thought I might have better luck compiling without OpenGL enabled. After I removed those lines from from hvirtual_config.h, I did a make clean;make. This time around, I was able to get Cinelerra 4 to start properly. Though, it soon locks up when viewing my 720P MPEG-TS files:
[mule@ogre cinelerra-4]$ ./bin/cinelerra
Cinelerra 4 (C)2008 Adam Williams

Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.
[mpeg2video @ 0xeafd00]slice mismatch
[mpeg2video @ 0xeafd00]mb incr damaged
[mpeg2video @ 0xeafd00]mb incr damaged
[mpeg2video @ 0xeafd00]invalid cbp at 14 37
[mpeg2video @ 0xeafd00]slice mismatch
[mpeg2video @ 0xeafd00]ac-tex damaged at 25 40
[mpeg2video @ 0xeafd00]ac-tex damaged at 6 41
[mpeg2video @ 0xeafd00]ac-tex damaged at 7 42

So much for that experiment! I'm going back to the CV version for now.
the mule

ffmpeg pipe to mpeg2enc

2009-05-18T10:50:00.017-04:00

Occasionally, I'll need to send a video stream into mpeg2enc. Mpeg2enc doesn't take an input file; it only accepts a yuv4mpeg stream. In order to send a yuv4mpeg stream to mpeg2enc, I do this using ffmpeg and the -f yuv4mpegpipe command line switch. Also, for best quality, I will send the stream using the FFMPEG variant of the Huffyuv lossless compression algorithm. ffyhuff is an enhanced version of Huffyuv that compresses better than Huffyuv.

Update 2009/05/19
As per Dan Dennedy's comment below, ffmpeg's yuv4mpegpipe command will ignore the -vcodec option and pipe the video stream to mpeg2enc using an uncompressed C420jpeg stream, which is an uncompressed YUV format. Certainly good enough for the likes of me!
*** end update ***

Here is a sample command to reencode a 720P video stream as a yuv4mpeg pipe to mpeg2enc:
ffmpeg -threads 4 -i INPUT.M2V -f yuv4mpegpipe - ¦ mpeg2enc --verbose 0 --multi-thread 4 --aspect 3 --format 3 --frame-rate 4 --video-bitrate 18300 --nonvideo-bitrate 384 --interlace-mode 0 --force-b-b-p --video-buffer 448 --video-norm n --keep-hf --no-constraints --sequence-header-every-gop --min-gop-size 6 --max-gop-size 6 -o OUTPUT.M2V

Note that I am taking advantage of the eight processors in my dual quad core using the multithread switches in the commands to both ffmpeg and mpeg2enc. Note that the eight threads have been split evenly, four to each encoder, to avoid CPU context switching. (Thanks again, Dan!)

Here's another trick: to see the header information of a YUV4MPEG stream, pipe the FFmpeg conversion stream to head -1 like so:
ffmpeg -i intermediate.mov -vcodec mpeg2video -f yuv4mpegpipe - | head -1

The FFmpeg output should show you some very important information, bolded below:
the output format: YUV4MPEG2 stream
height and width: 1280x720
framerate: 30001:1001 (or 29.97fps)
colorspace: C420JPEG
not sure what IP: 1 or XYSCSS is

Duration: 01:19:46.74, start: 0.000000, bitrate: 110301 kb/s
Stream #0.0(eng): Video: mjpeg, yuvj420p, 1280x720 [PAR 1:1 DAR 16:9], 108762 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc
Stream #0.1(eng): Audio: pcm_s16be, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, yuv4mpegpipe, to 'pipe:':
Metadata:
encoder : Lavf52.64.2
Stream #0.0(eng): Video: mpeg2video, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0.0 -> #0.0
Press [q] to stop encoding
YUV4MPEG2 W1280 H720 F30000:1001 Ip A1:1 C420jpeg XYSCSS=420JPEG

Sweet, eh?

As a final note, I am a bit confused on the differences between FFMPEG compression algorithms: ffyhuff and ffv1. If someone has pointers to the documentation on these, I'd be interested in finding out more. A google search just added to my confusion.

the mule

References
mpeg2enc man page
mpeg2enc manual
ffmpeg vs mpeg2enc
Huffyuv
FFV1
FFMPEG How To

related posts
http://crazedmuleproductions.blogspot.com/2010/01/batch-render-redux.html
/2010/01/compile-times-performance-improved.html
FFMPEG HowTo

VMware virtual appliance for video editing

2009-05-13T11:18:00.002-04:00

Over the weekend, I created a VMware Partner Account and got my Fedora 10, x86-64 virtual machine approved to be listed on VMware's Virtual Appliance listings:
http://www.vmware.com/appliances/directory/148183

If you want to try out Cinelerra and you use 64-bit VMware Player, Workstation or Server, this is an easy way to get started. I'd appreciate someone giving it a shot and letting me know how it works.

the mule