forums.etree.org

Hiya Linux Server Geeks-

I've been having issues with parsing when doing mp3 creation from flac.

the command I am running is basically this:

/usr/bin/shn2mp3 -b $bitrate --target \
$path/friends/$foo/mp3/ \
$path/friends/$foo/flac/*;

an example of my issue:

the text file:
--------------

John Hartford
Venue: Unknown
Asheville, NC
3/31/1993

Source: SBD>DAT

Transfer: D8>Zefiro ZA2>Soundforge 4.5 XP>CDWav>CDRWin>CDx?>EAC>SHN DAT provided by Jeffrey Mankin (but I got it from Ken "Easy Ed"$

set 1 disk 1:

01 Intro
02 The Girl I Left Behind Me (banjo)
03 Takin' Me Back To My Mississippi River Home (banjo)
04 Fiddle Tune ?>
05 Billy in the Lowground (fiddle)
06 Hounddog (fiddle)
07 Back in the Good Old Days (fiddle)
08 Silvertone Guitar Blues (guitar)
09 In Tall Buildings (guitar)
10 Gentle on My Mind (banjo)
11 John Henry is Dead and Gone (banjo)
12 Can't Stand to Throw Anything Away (fiddle)
13 "Have You Ever Thought About Playing Detroit City?"
14 When They Hear Mother Praying So Loud in Her Sleep (banjo)
15 *Forked Deer> (fiddle & *banjo)
16 *Mouth Jam> (mouth & *banjo)
17 *Forked Deer (fiddle & *banjo)

set 2 disk 2:

01 Intro
02 Up on the Hill (banjo)
03 Here I am in Love Again (banjo)
04 Bring Your Clothes Back Home (fiddle)
05 The Golden Eagle Hornpipe> (fiddle)
06 Natchez Under the Hill> (fiddle)
I Love A Little Girl With Hair Down Behind (fiddle)
07 On a Long Hot Summer Day (fiddle)
08 Don't Leave Your Records in the Sun (banjo)
09 Banjo Tune ?
10 How It Happened > (guitar)
11 Presbyterian Guitar (guitar)
12 Going Back to Dixie (banjo)


Parse output: (path) replaces actual path:

Directory: (path)/friends/Hartford/flac/hartford1993-03-21.flac16
Info File: hartford1993-03-21.txt
Output Dir:(path)/friends/Hartford/mp3/hartford1993-03-21.mp3f
Band: John Hartford
Date: 1993-03-31
Venue: Venue: Unknown Asheville, NC
Source: Source: SBD>DAT Transfer: D8>Zefiro ZA2>Soundforge 4.5 XP
CDWav>CDRWin>CDx?>EAC>SHN DAT provided by Jeffrey
Mankin (but I got it from Ken "Easy Ed" Edwards)
Transferred and Mastered by: Brad Hefner
(gg_dd_aa_ee@yahoo.com) 01 Intro
02 The Girl I Left Behi ...

Track Time Title / Comments
------- ----- ----------------
1/17 0:00 Intro
File size 739.2 kB
2/17 0:00 Up on the Hill (banjo)
File size 2.9 MB
3/17 0:00 Here I am in Love Again (banjo)

The Parse Problem:
-----------------------

I've reformatted the text file a dozen different ways, but set 1 is included in the source info, and set 2 is the only info parsed as tracks, so the first mp3 is tagged with the info from set2, t1.


The Flacify Problem:
------------------------

Also, Flacify is broken when going from wav>flac. We have some folks upload in shn and for consistency we use flac only. Flacify does the shn>wav portion of the process fine, but, even after re-installing flac and the etree scripts, doing updates on all of the perl modules, the issues still exist.

the error in the flacify script makes no sense for the line number the error is supposedly from.

Jul 21 11:24:18: INFO: Converting WAV files to FLAC
Modification of a read-only value attempted at /usr/bin/flacify line 241, here is that section of code:

241 map {
242 my $set = $_->{Set} || 0;
243 $GROUPS{$set}->{$_->{Index}} = $_;
244 } @$songs;

Any help? I'm noticing this as we rebuild after a drive failure (though the problem existed previously as well, I just worked around by downloading, using mkw to shn>wav and flac front end to create the flacs, then reupload, which -sucks-.

This is for a production server for a non-profit in use at www.thespps.org - any help would be appreciated.

-Paul
e mail to prennix -at- gmail d-o-t c-o-m

well im no linux server geek
but i do use linux & my wife says i geek it up sometimes but ive yet to snort or make any odd noises when i laugh
im guessing you change bitrate on all files id say edit "/usr/bin/shn2mp3" & enter bitrate you want
as for tagging i on occasion come across the same issue on some shows it isnt often though, & dont know why i edit txt file 100x with no success

as for flacify problem

ill bet you copied these files from a cd to hard drive in which when writing to cd the permissions are set to read only. if you change permissions that owner can read/write will fix this issue. i might lose this bet but its what id be looking at

btw what distro you use, out of curiosity?
archlinux here

"
im guessing you change bitrate on all files id say edit "/usr/bin/shn2mp3" & enter bitrate you want
Code:
my $bitrate = 192; <<line 47
mine
my $bitrate = 256;
"

I use a script that works through various directories and creates mp3s. The bitrate is declared in the script. this isn't the problem though ; )

"
as for tagging i on occasion come across the same issue on some shows it isnt often though, & dont know why i edit txt file 100x with no success
"

yea, that is pretty much the same thing i am experiencing.

"
as for flacify problem
Code:
Modification of a read-only value attempted

ill bet you copied these files from a cd to hard drive in which when writing to cd the permissions are set to read only. if you change permissions that owner can read/write will fix this issue. i might lose this bet but its what id be looking at
"

tempting to think this, but I reset all permissions and ownership on the server. this is a remote box, nothing is done locally, so on upload the files all have the same permissions.

The flacify will complete the shn>wav step, but it pukes on the beginning of the wav>flac everytime.

btw what distro you use, out of curiosity?
archlinux here

CentOS

Hi. I'm the author of etree-scripts, so hopefully I can help :-)

The problem is the word "Stand" in the Set 1 track list is causing that stanza to be mistakenly identified as source info. Just FYI you can peek under the covers of flacify a bit by passing it the --debug flag to see what the parser thinks it is seeing.

This turns out to be a bug in the Etree::InfoFile parsing module, so it would manifest in both 'flacify' and 'shn2mp3' though perhaps in slightly different ways.

Thankfully I seem to have fixed this bug, though I neglected to actually release it :-( See if replacing your Etree/InfoFile.pm with the version at http://bklyn.org/~cae/InfoFile.pm fixes this problem.



The diagnostic is confusing but correct. Basically, if you refer to $hash->{Key} and that key does not exist, Perl will create it with an empty value for you. But $_ is read-only, so that operation generates this diagnostic. What this means is that the internal data structure used by flacify to keep track of the files is missing either a 'Set' or an 'Index' key.

I'm not certain, but I suspect this is more of the same bug from shn2mp3 above. Was this the same show you were working on? If so, hopefully the InfoFile.pm fix will work here too.

Please let me know if this is the case, and I'll make a (long overdue) release of the etree-scripts with this and other fixes.

Thanks so much.

I am encouraged that you are still maintaining these scripts, to say the least! Thanks so much for the help, we are getting somewhere.

I replaced InfoFile.pm, and it fixed the flacify issue, but not the parse issue.

Debug output (or the first 100 lines covers it) - again, thanks so much for your help!

# cat debugout | head -n100
FRIENDS
>BAND/VENUE/DATE<: John Hartford
March 21, 1993
Unknown Venue
Asheville, NC
>BAND<: John Hartford
>DATE<: March 21, 1993
>VENUE<: Unknown Venue
>VENUE<: Asheville, NC
>SOURCEINFO<: Source:SBD>DAT
Transfer: D8>Zefiro ZA2>Soundforge 4.5 XP>CDWav>CDRWin>CDx?>EAC>SHN
DAT provided by Jeffrey Mankin (but I got it from Ken "Easy Ed" Edwards)
Transferred and Mastered by: Brad Hefner (gg_dd_aa_ee@yahoo.com) 10.15.02
>SOURCEINFO<: Set I
1. Intro
2. The Girl I Left Behind Me (banjo)
3. Takin' Me Back To My Mississippi River Home (banjo)
4. Fiddle Tune ?>
5. Billy in the Lowground (fiddle)
6. Hounddog (fiddle)
7. Back in the Good Old Days (fiddle)
8. Silvertone Guitar Blues (guitar)
9. In Tall Buildings (guitar)
10. Gentle on My Mind (banjo)
11. John Henry is Dead and Gone (banjo)
12. Can't Stand to Throw Anything Away (fiddle)
13. "Have You Ever Thought About Playing Detroit City?"
14. When They Hear Mother Praying So Loud in Her Sleep (banjo)
15. *Forked Deer> (fiddle & *banjo)
16. *Mouth Jam> (mouth & *banjo)
17. *Forked Deer (fiddle & *banjo)
>DISCINFO/TRACKINFO<: Set II
1. Intro
2. Up on the Hill (banjo)
3. Here I am in Love Again (banjo)
4. Bring Your Clothes Back Home (fiddle)
5. The Golden Eagle Hornpipe> (fiddle)
6. Natchez Under the Hill> (fiddle)
I Love A Little Girl With Hair Down Behind (fiddle)
7. On a Long Hot Summer Day (fiddle)
8. Don't Leave Your Records in the Sun (banjo)
9. Banjo Tune ?
10. How It Happened > (guitar)
11. Presbyterian Guitar (guitar)
12. Going Back to Dixie (banjo)
13. Deep River Blues (banjo)
14. Vamp in the Middle (fiddle)
15. *Bonapart's Retreat (fiddle & *banjo)
16. *Square Dance Call (fiddle & *banjo)
17. *Tennessee/Asheville Waltz (fiddle & *banjo)
>SET 2< Set II
>TRACK 1< 1. Intro
>TRACK 2< 2. Up on the Hill (banjo)
>TRACK 3< 3. Here I am in Love Again (banjo)
>TRACK 4< 4. Bring Your Clothes Back Home (fiddle)
>TRACK 5< 5. The Golden Eagle Hornpipe> (fiddle)
>TRACK 6< 6. Natchez Under the Hill> (fiddle)
>TRACK 7< I Love A Little Girl With Hair Down Behind (fiddle)
>TRACK 8< 7. On a Long Hot Summer Day (fiddle)
>TRACK 9< 8. Don't Leave Your Records in the Sun (banjo)
>TRACK 10< 9. Banjo Tune ?
>TRACK 11< 10. How It Happened > (guitar)
>TRACK 12< 11. Presbyterian Guitar (guitar)
>TRACK 13< 12. Going Back to Dixie (banjo)
>TRACK 14< 13. Deep River Blues (banjo)
>TRACK 15< 14. Vamp in the Middle (fiddle)
>TRACK 16< 15. *Bonapart's Retreat (fiddle & *banjo)
>TRACK 17< 16. *Square Dance Call (fiddle & *banjo)
>TRACK 18< 17. *Tennessee/Asheville Waltz (fiddle & *banjo)
>ETC< * with David Holt on gutstring banjo
Etree::InfoFile dump:
$VAR1 = bless( {
'Band' => 'John Hartford',
'ByExt' => {
'shn' => {
'hartford1993-03-21d2/hartford1993-03-21d2t11.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t08.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t14.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t05.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t06.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t02.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t08.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t17.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t12.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t14.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t07.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t16.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t11.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t15.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t04.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t02.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t10.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t05.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t01.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t15.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t12.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t07.shn' => 1,
'hartford1993-03-21d2/hartford1993-03-21d2t17.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t16.shn' => 1,
'hartford1993-03-21d1/hartford1993-03-21d1t06.shn' => 1,

Hm, OK close but no cigar. Which script is this the output from?

This is odd because I tested this input data with the new InfoFile.pm and it parses the first disc correctly. Are you sure you replaced/overwrite the right InfoFile.pm in your install?

You can verify where Perl (and ergo shn2mp3/flacify) is picking it up from like so (cut & paste into your shell as the quoting is kindof nasty):



That is the file you'll need to overwrite with the new one from http://bklyn.org/~cae/InfoFile.pm. Also, just to be certain, the "Id" stamp in this file should read:



I made some very small changes (nothing that should affect this) in the past 30 mins or so that might help debug this if it still doesn't work, so please re-grab http://bklyn.org/~cae/InfoFile.pm and update your copy.

yup. I replaced the wrong file. Thanks SO MUCH.

The interesting this is, replacing the version in the source directory, which is what I did the first time, fixed flacify. (?)

I really appreciate your help, and look forward to future releases - if you have an announce list for updates, please put me on it.

Thanks Again,

Paul

another thing I am noticing (now that I turned on the --force flag to rewrite all of our mp3s)..

Is that If the date is unknown, it will again add the first set to the source.

(I am not a Perl guy)

What would be a good way to force it to recognize the beginning of the track list, instead of the end of the source?

I'll take a look through that infofile.pm tonight (thanks for the good notes in the code!) Perhaps I can come up with something that will work for our odd archival work, which isn't always completely relevant to the live music trading community.

Glad it finally worked. The reason flacify worked is because (mostly for my own testing) it looks for a local copy of Etree::InfoFile before a globally installed one ('use lib dirname $0'). I should probably remove that.



I'm pretty irregular with the releases, truth be told, but I expect to do a 3.2 release as a follow up to this. I've been hitting similar bugs myself and had just not gotten around to fixing them. A little complaining from a real user, and I'll get off my duff!

I think you can use features of Sourceforge to "watch" different projects, especially if you have an RSS aggregator. For example, the "downloads" section has an RSS feed:

http://sourceforge.net/export/rss2_proj ... p_id=60296

There is also the (similarly infrequently updated) blog-style home page:

http://etree-scripts.sourceforge.net/

Though I'm in the process of revamping that to use a different blogging engine. At any rate, it will have an RSS feed too.

Meaning this is a file set from an unknown date, or just that the code was unable to parse the date from the text file correctly? In the latter case the fix is simply to re-format the dates in a way that Etree::InfoFile will understand (yyyy-mm-dd is my personal favorite for ease of parsing and sorting). In the former case, its a little trickier.

I've basically written the code with the assumption that a file set is for one-and-only-one well defined date. But I realize that this is not always the case, for example compilations (e.g. multiple dates) or recordings of unknown provenance (e.g. no date).

All this said, I think the work-around is simple. Simply comment out line 371 in InfoFile.pm. Here's the relevant section with an added comment:



I think that will work, but I'm not 100% sure what the follow-on effect will be to shn2mp3 and flacify.

I'm still tweaking the code, and you can grab the latest from the Sourceforge SVN repository (http://sourceforge.net/svn/?group_id=60296) but I hope to do an official release this week.

Hello,

Sorry for exhuming an old old thread, but I was wondering if the official release of these updates ever happened. I'm about to get a massive pile of Dead shows dumped on me, many in .shn. I'd like to be able to auto-convert them to flac, but the last time I tried these scripts I was experiencing similar problems to Paul.

I clicked some links but it looks like the most recent .deb files are from Version 3.1 in '05. I'm not much for compiling my own software, though I suppose I could give it a try. But if you have binaries done (suitable for use in Ubuntu) that would be excellent.

While I'm typing a wishlist, having a GUI for this would be, like, totally awesome :-D

Thanks!

I think that the .debs have been updated, but I'm not sure. Only one way to find out . . . .

I dunno, it looks like the latest version is 3.1, updated July '05.

try this:

http://phrog.etree.org/software-linux/e ... pts.tar.gz

Hmmm... that whole domain (phrog.etree.org) is unavailable

seems fine from here.

I'm getting a "taking too long to respond" error. When I put a www. in there it just fails entirely.

Could just be slow networks or something. I'll try again periodically throughut the weekend.
thanks for pointing me here.

OK I can access your site now

I d/l'd the scripts and I think I installed them properly, thanks!

Hey just a reply to say that this new script seems to be working pretty good

Occasionally need to tweak the text file to get it to parse right, but so far nothing like it was in the earlier version.

Thanks!

Just wondering about getting some updated builds on sourceforge. I looked last night, and the only versions there are still 3.1-x from 2005. Caleb?

anybody know where to find caleb? looks like bklyn.org has dropped off the face of the earth.

I'm involved with a few large projects to get every show on hard drive for both GD and phish, and there are a lot of people chomping at the bit to tag these.

if you could wrap up 3.2 in a self-installing cygwin version, that would be awesome!

thanks for all your hard work!

Will these work in cygwin?
http://phrog.etree.org/software-linux/e ... pts.tar.gz

i think so, but its been a while. i'm just trying to simplify for people who can help tag who arent into linux.

heres the instructions i had written 3 years ago, and just dug up from a old email. Shoud work with the current version as well:

ok, got it worked out. make sure you d/l the 'flacify_here.reg' from my site it is updated to run 'flacify *.*' instead of 'flacify'


1. Install Cygwin

go to http://www.cygwin.com/

click on the 'install or upgrade now' icon to download 'setup.exe'

run setup.exe and follow the prompts

select 'install from internet'

for root directory select 'C:\Cygwin'

for local package directory - make a folder and path that you can find, this installer basically d/l's all the packages you need here. You wont need this after install unless you want to install more cygwin packages for different uses later.

select 'direct connection', pick an ftp site.

under select packages, you want to use the default, except for the following addition: Go to 'interpreters' and select 'perl'

let it install. It may give you a message that an older version of cygwin has been found (usually for shntool). You can let it delete the old file.

2. download etree-scripts-3.1.cygwin.exe from http://sourceforge.net/projects/etree-scripts/ and run it, when it asks you for the directory you want to unpack to, select C:\Cygwin\

4. download 'flacify-here.reg', 'cygwin-here.reg', and 'turn_off_read_only.reg' from www.simplefuels.com/tapergeek Double-click them to install them to your registry.

ok, now youre all set! In order for Flacify to work correctly, it must be run from a top-level folder containing folders of shn or FLAC files, not on the shn or FLAC folders themselves. This is mainly because it renames the folder from .shnf to .flac16 when done, and it cant do this if it that directory has an active shell running in it. It is best to dump the folders for all the shows you want to process into a folder called 'to be flacified' or something like that.


D:\
-to be flacified
-ph1998-08.01.shnf
-gd1977-05-21.flac16


5. right click on the 'to be flacified' folder containing .shn or .FLAC folders, and select 'flacify here'. Since this is your first time using Cygwin, it will auto-configure a few 'skeleton files', then run the script.



with 'flacify here',the script will run, but the window will close when its done. If something went wrong, you wont see the error message. To debug, use 'bash shell here' instead , then type 'flacify *.*' or 'flacify *.shnf', etc.

some linux101:

basic linux (not really required when using the reg entries above, but good to know):


you need to mount a drive or folder for cygwin to see it. This is done by creating a directory, and mounting the filesystem to it. You can mount a whole drive, or just a folder to it

So say your files are in D: drive, open cygwin (doesnt matter where)

mkdir /d
(make a linux folder named d (the '/' is important, as it takes you to root instead of a relative path in your home dir.)

mount d: /d/
(mounts your d drive to that folder)

cd /d
(goes to that folder)

ls
(check youre in the right place by doing a directory listing)


currently it hangs on renaming the folders and stops the process if they are read only. use the read-only.reg entry at my website first. theoretically it should process all the shnf folders and rename em to flac16 if successful

so an example, you have a folder called 'doneshns' on your e drive, that has *.shnf subdirectories in it containing shows. you are gonna create a linux folder called 'toflacify', mount the folder there, and flacify 'em:
mkdir /toflacify
mount e:/doneshns /toflacify
cd /toflacify
flacify *.shnf

Flacify works 99% of the time for me when I go in and make sure the text file starts like this:

-------------------
Phish
1997-07-31 (or 7/31/97 or July 31,97, etc)
Shoreline Amphitheatre - Mountain View, CA

and the tracks have #'s

It then parses this file to make Artist=Phish
Date = 2004-07-31
Album = Shoreline Amphitheatre - Mountain View, CA
Source = etc...

then names all the songs.
then takes comments marked with '*', etc and applies em as comments to the tracks

For me, the key is to make it "Phish", not phish or PHiSH or PHISH. And to make the 3rd line 'venue - City, ST'

kinda being picky, but the point is if you take a little extra time, you save tagging on the backend, and keep your collection consistent.

Also, if the artist is missing from line one, it freaks out

then I use godfather or dbpower amp to add 'encoded by', genre=jamband, and to bulk-edit any other tags if needed. sometimes the songs are not right, so you need to check and make sure the last track of the show is correct, and youre good to go.

sounds difficult, I know, but really a breeze once you get it.

As Caleb has mentioned elsewhere, he has updated the scripts, providing several fixes. Unfortunately, he has not uploaded the newer branch (3.2) to the stable at sourceforge. In Caleb's post here (http://etree-scripts.sourceforge.net/), he mentions that the updates are available via subversion. Well, I use CYGWIN myself, but had never used subversion before, so I thought I'd share the full process for installing the scripts under CYGWIN a la subversion...

1. Open bash prompt under CYGWIN
2. Install CPAN
% cpan
(Answer setup questions along the way, it will ask for paths to some standard packages like make, ftp, ncftp, unzip, tar, etc... If you need something just rerun the CYGWIN setup and install it)
3. Install Subversion under CYGWIN (using the installer)
4. Under CYGWIN, create your repository
% mkdir svnroot
% svnadmin create svnroot/etree-scripts
% cd svnroot
% svn co https://etree-scripts.svn.sourceforge.n ... ee-scripts etree-scripts
5. This will download and install the files in the svn tree. Now, you should be able to move into the trunk/etree-scripts directory and see the current release files. There is (thankfully) a README and INSTALL doc, outlining the requirements and install procedure, which is very simple. I did not find the CYGWIN auto installer in the trunk, but compilation with perl went without a hitch.

I hope that's clear. Maybe it will will help someone...

Andrew

NOTE: Looks like Caleb never updated the InfoFile.pm in SVN according to the changes he mentions earlier in this thread. I thought about posting a modified copy, but I don't want to leave a copy somewhere that might confuse if Caleb does further work... so you'll have to update your existing copy.