forums.etree.org

Ever since the LivePhish Download series came out with SBEs there seems to be alot of shows appearing to have SBE's (sector boundary errors).

I have noticed that most of the SHNs i have been getting have at last 1 SBE (and the MD5s match whats on etree, thus the original have SBEs) , is this something that we are gonna have to start checking ALL SHNs in circulation. Also I have discovered that all SHNs before 10/2000 are not seekable. Looks like a lot of people are gonna be doing SHN > WAV> shntool (fix SBEs) >SHN (seektable appended) re-burning, then all the new MD5s are gonna have to be posted with new sources, sounds like a nightmare is about to take place.

And whats worse, new SHNs coming into circulation seem to have them too!

OPINIONS are welcome


-just my 2 cents

as a side note for everyone, Furieo! corrects SBEs when burning an audio CD, FYI for everyone

You can easily check for SBEs in your whole collection by running shntool, which you can set up for a right-click check, for example
http://research.umbc.edu/~hamilton/cgreenshntool.html

Just pop your shn archives discs in and out of the computer. Feel free to add public comments abot SBEs to any existing shn record in the sources in circ db.
http://db.etree.org/shncirc

SBE fixes can be done by volunteers and "reseeded" as they desire. As long as they give people a heads up- rewrite txt files, for instance- these new versions would be welcome int he trading pool and as new db entries.

BTW Don't seek-append nonseeking shns unless you're making some other fix (like SBE or tracking) and therefore issuing "new" shns. If you want seeking, you can generate separate seek tables for yourself. Simple reshn with seek-appending is where "version tracking" problems become difficult.
http://www.etree.org/shnv3faq.html

If you see new-release shns with SBEs, feel free to politely educate the seeder, perhaps pointing them to some guidelines:
http://wiki.etree.org/index.php?page=SeedingGuidelines

SBEs can be harmless (though the first batch of Live Phish ones were generally not). If they come on the last track of a disc, or are followed by a track which begins in silence (e.g. end of Set I followed by start of Set II which has a fade in), the null samples which will be added to the end of the track on your audio CDs will be inaudible and harmless. If they pop up in the middle of a set or on EVERY track, then they suck


IMHO it is in part the responsibility of etree.org siteops to check a seed for SBEs before putting it up for public hosting. Siteops should use shntool to fix a seed, rename the directory to contain "-fixed" and then, and ONLY then, host the show publicly. The errors should also be communicated back to the seeder and/or uploader of the show.


SHNv3 (seekable SHN) didn't really exist before 10/2000, which would explain what you see. Its is not necessary to change a SHN to make it seekable. You can use the command line shorten program's -s flag to make separate Seek Table (.SKT) files which will be honored by most SHN plugins.


Indeed, too many people who seed shows don't bother to read the Seeding Guidelines. Its not hard. Just cut on sector boundaries.

By coincidence I wrote a couple posts on this over the last few days for antsmarching.org, but they seem to fit right in here:

SECTOR BOUNDARY ERRORS

It's best to check your shows for "sector boundary errors" (SBEs) before uploading any files. A sector boundary error results when a wav file is not the right length to exactly fill a CD sector (= 1/75th second). Most burning programs will fill the balance of the sector with silence, sometimes creating a microgap. Some burning programs will drop the partial sector altogether. For a little more discussion of SBEs, see etree's Seeding Guidelines.

SBEs usually result from a wrong setting in an audio editor when the DAT > WAV conversion is done. For example, in Cool Edit, if it's not set to Snap To Frames, it will cut the wav anywhere, not just on a sector boundary. Tracking done using CD Wave will not create SBEs.

You can check for SBEs with shntool. See the ReadMe and Tutorial files, at that link and in the zip package, for more explanation. SBEs can also be repaired reasonably easy with shntool - it just moves a little bit of each track to fill out enough for a CD sector. But be sure to preview the repair because the tracks MUST be listed in the right track sequence to repair it properly!

And of course, the md5s will change if the SBEs are repaired. md5 summer is handy for recomputing them.

There is a nearly identical functionality in flac itself to repair SBEs. shntool can also be used to check flac files for SBEs.

You'll want to have shntool, shorten and flac in your system folder (or in a folder that is in your Path) for these commands to work from any folder. You also need cygwin1.dll, which is in the zip packages.

Here are some batch files / command lines for use with shntool, the items in brackets will vary depending on the files and the user! Whether you use a separate directory is up to you. Whether you need the "no skip" option depends primarily on whether the existing shns have seek tables or not, and whether you know how to add seek tables to any files that are missing them without reprocessing the files. (See below for info on that.) For more information on batch files, see this link.

To check for SBE's, copy this into a txt file called shnlen.bat:

@echo off
shntool len *.shn > len.txt
start notepad len.txt

To preview what shntool fix will do, copy this into a txt file called shnfix-preview.bat. This assumes the files have names that sort in track order, otherwise you have to specify the order.

@echo off
shntool fix -o shn [-d directory] [-noskip] -p *.shn > shnfix-preview.txt
start notepad shnfix-preview.txt

To use shntool fix, copy this into a txt file called shnfix.bat. This assumes the files have names that sort in track order!

shntool fix -o shn [-d directory] [-noskip] *.shn

SEEK TABLES

You can use shorten and some batch files to

- check to see if a file has seek tables
- append seek tables
- remove seek tables

without going through encoding and decoding them, which can be useful if you are seeding or reseeding a show. See this link for the latest version of shorten and this discussion of batch files But bottom line, people shouldn't be altering the shows. Just live with the way they are seeded. [See the other posts in this thread!]

Sample batch files, see the link above to help explain:

To see if the shns have seek tables

for %%f in (*.shn) do shorten -i "%%f"
pause

To append seek tables

for %%f in (*.shn) do shorten -k "%%f"
pause

To remove seek tables

for %%f in (*.shn) do shorten -e "%%f"
pause

SHNCLICK

shnclick "is a windows right-click context menu frontend for shntool. It is solely intended to provide quick access to common shntool tasks like checking and fixing sector boundaries on shorten files or stripping excess header info before seeding. ... More advanced useage will require using the command line." Or even better, batch files.

MAKING COMMANDS AVAILABLE FROM THE SHELL (Right-Click)

You can make these commands available by right-clicking on one of the shn or flac files in a folder by doing something like the following. This is how it's done in Windows 2000, presumably XP is similar - if someone can add a discusion of the specifics for XP, that might help:

Open any folder, such as My Computer. Choose Tools, Folder Options, File Types. Find shn (or flac). Click Advanced. That gives you a list of the choice of options that will appear when you right-click on that file type.

Click on New. Under Action, type the words "Create len file" (without the quotes). In Application used to perform action, type something like:

C:\Utilities\shnlen.bat "%1"

using the location and name of the batch file that performs the appropriate action - in this example, the file discussed above called "shnlen.bat" in the C:\Utilities folder which runs the shntool len command and puts the contents into a file (which you can keep with the set of shns to show file sizes, times and whether there are errors in the files). Then click OK.

You can repeat this to create multiple right-click menu choices for shn or flac files. For example, "Check for seek tables." If you've got mkwACT and WinAmp installed, "Decode to wav" and "Enqueue" are among the actions already listed here; using Edit you can see how those call the respective programs.

Someone could probably easily write a script to install the batch files and add the shell extensions (ie, the right click features). Not me, unfortunately ...

Hey, since this is turning into a nice meaty thread (thanks for linking it in the guidelines, buzzy!), I'll throw in this bit of wisdom I learned from Chris Ladner:

If you are using shntool to fix SBEs on a group of shns, group and fix them by [musical] set, not by disc. That will make rearranging tracks across discs, or moving to longer media, a much happier experience for people.

Plus the other nice thing.....If you correct for each set, even if you have to split a set across 2 discs, the chances of having any SBE's is almost nill.....

That's why a couple of us started that project to correct all the SSSB's....we got fed up with the lackluster seeding of those specific shows.....I mean really, who here uses only 74 min CD's?

It's almost enough of a problem nowadays to start a quality control group to check just for these things before a show gets added to the db.etree......

Marmar

So just to point out a subtlety that escaped me the first time, set in this case means "musical set" not "set o' shns" - in which case doing it by set works.

<obligitory plug>
...or, just seed your .wav files using FLAC, and make sure the "Align on Sector Boundaries" option is checked (or for you commandline guys, use the --sector-align flag).
</obligitory plug>

If I take a SHN > WAV > FLAC and use the option to Align on Sector Boundaries in FLAC Frontend while encoding, upon the subsequent decoding back from FLAC > WAV, will my resulting WAVs be any different i.e.) are they now free of any Sector Boundary Errors (SBEs)?

Is verifying the resulting WAVs with the original wav.md5 enough to tell if errors have been fixed? --Assuming my MD5s verify has anything really changed? Or should I expect the new WAVs not to match the old ones if FLAC has fixed any SBEs?

Is the following is taken from:

http://wiki.etree.org/index.php?page=FlacFrontend

Advanced: Fixing Sector Boundaries
Important: Do not try to use the Sector Boundary alignment feature with FLAC Frontend. The way FLAC Frontend passes information to FLAC breaks the --sector-align functionality (all your misaligned tracks will be padded with zeros, rather than really fixing it!!!). This is a known bug with FLAC Frontend, and not with FLAC itself.

--Is this really a 'bug' and has it been 'fixed' in the updated release of FLAC Frontend?

buzzy the batch files are very helpful thanks a million

BKLynn mentioned that SBEs may not be a problem at the end of discs. I disagree with this only because of the possibility of PCM WAV burning to a longer media like DVDR.

If you string together several shows, each having an SBE at the end of the last track, your new longer disc will have pops between the end of one show and the beginning on the next.

Now, the new technology is not out and common yet, and it may very well change and my point may be moot. But I think its a good idea to fix the SBE at the end of the last track also.

But no SBEs on the last track is especially important if the the set is cut between 2 discs. The last track of disc 1 MUST be SBE free so that it will align with the first track of disc 2. Let's say I decide to burn an "excerpts" disc and want to have the end of disc 1 and the beginning of disc 2 together, I don't want an SBE in the middle of my disc. I also don't want any kind of blank space between the two tracks (however small), so I cut the tracks/fix the SBEs by set rather than by disc. That way the padding is added at the end of the last song of the set, rather than between 2 songs. I actually would like to see NO SBEs and NO padding, the whole recording should be cut to flow without SBEs and the beginning AND end should be trimmed. Like I said, potential of 3 hour discs with DVDA or somthing else...

And I also mentioned "set". Yes, this means musical set (as in set 1, set 2, encore) rather than a set of shns. Though, I think if you fixed the entire SHN set all at once (so that the padding went on the last track of the entire show), it fine too. Doing it this way prevent silence/padding between set 1 and set 2, which may be a problem if set 1 ends on the same disc as set 2 begins, or on longer media.

SBEs I find occur mostly on WAVs not tracked with CDWave (check the text file). You'll see all kinds of WAV editors being used, and even Cool Edit can have problems. I wish everybody would use CDWave - its so damned easy!

As for reSHNing non-skt SHNs (SHN>WAV>SHN w/ seeks), I think as long as you document in the text file, its not a problem (I'll go back and read the link Diane). Anyways, doesn't the new version of SHN compress better than older versions??? Couldn't a 2 disc SHN set be reSHNed to only take up 1 disc???

While I'm not a fan of SHN/FLAC>WAV>SHN/FLAC in general (yes, more version is bad), I do think that people should feel free to improve existing SHN sets. As long as proper notation is maintained in the text file, it shouldn't be a big deal.

And while I a fan of FLAC, I'm not fully aware of all its functions etc., so I don't use it that much. And thanks for the wiki.etree find there Samurai, could you address this Mike Wren??? If FLAC is not fixing SBEs properly, it should be fixed, or the function removed - it'll cause worse problems to have padding...

Which brings up another question. How do you fix WAVs with zero padding at the end. To SHNTOOLs, it looks fine, but to us, it has a audible blank spot (for about 0.15 seconds - sounds like a pop actually). I've been using SF to trim the blank spots, but sometimes music doesn't line up correctly (millisecond missing). Can you use EACs remove trailing silence to remove these???

Terry

PS Nice thread! One with real substance!

I'm not sure which new version you might be referring to - but in general a given wav will always recompress to exactly the same shn with the same md5 (if the seek table setting is the same) - so that implies no difference in compression.


Yes, you're right - a file with embedded silence at the end will not show as having SBEs or other errors when using shntool. (On a studio album, silence is there intentionally.)

One possible explanation for how these come about is that they were files with SBEs which got burned to CDR, thereby embedding that silence into the track. That is, the reason you can hear microgaps on some tracks is because it embeds a partial sector of silence into the track.

But that would only be the explanation if the silence is less than 1 sector = 1/75th second = 0.01333 seconds. If you're seeing 0.15 seconds, that's much longer than could be explained by SBEs resulting from the way an audio editor was used.

As far as fixing them - it depends somewhat on the use. If it's in the original seed (because for example the taper did the editing, with SBEs, and burned to CDR for someone else to shn - which I have seen happen) - then ideally you'd use an audio editor to fairly precisely trim the embedded silence, then use shntool to fix the SBEs that will likely create.

EAC's remove digital silence feature may work - the only reservation is not knowing just what it's doing to each track. So for that reason, I'm not sure I'd use it for a seed. But if it's for a quick-and-dirty fix for your own use or for CD audio copies (vs shn seeds), you might use the trim silence feature; then fix the SBEs before burning. (If you don't fix the SBEs, of course, it will recreate some gaps.) That might give a reasonably easy fix that's fine for that use.

It's also usually worth thinking about how that flaw got into the show, sometimes it's trying to tell you something. For example, there are lots of shows out there that have microgaps because they are mp3-sourced, not because they have SBEs - so it's worth trying to rule out an mp3 source. And in general SBEs only introduce silence at the end of a track (though it might move a little if it's burned and ripped more than once) - so if you're seeing silence at the beginning of a track, again check for a compressed source.

Or, if you think you'll be sharing a lot of copies of a show, it may suggest trying to verify that you have a copy of the best available source.

There isn't really much choice, when you come down to it. Ideally, the last track of each disc/show would peter out to silence and end on the sector boundary, but if it doesn't, the only thing you can do about it is fade out the music and pad the end with zeros . . . which is what you'd wind up with anyway if the job had been done right the first time.

Well, actually, it's a little better than that. You'd have some background noise instead of all zeros, which would create an abrupt transition to the digital silence at the begining of the next disc/fileset/whatever comes next.

IMHO, the difference is so minimal that no one would notice the difference unless she was listening for it.

What if anything is really wrong with using FLAC Frontend to -align on sector boundaries? Can you explain easily the difference between how Frontend uses padding with zeros and why some people say that this is not the same as how shntools fixes the SBE's? Is the 'difference' noticable to the ear?

This a very informative thread, thanks for all the info. Using shntool with Linux on the command line is pretty easy so I find no need for scripts to automate it at all. There is an issue with shorten generating skt files or appending seek tables though, it won't take a * wildcard varible so you can't just do: shorten -s *.shn and you have to do each file separately by hand. Here are some scripts to make this easier:

The first one will generate separate skt files for all .shn files in a directory. I call it 'sktgen' but you can call it what you like.

#!/bin/bash

for shnfile in *.shn ; do
shorten -s "$shnfile"
done

chmod +x and make sure it is in your path then just cd to a directory with non-seekable shn files and type 'sktgen'. Do the same with the following:

to test if the shn files have seek tables appended:

sktcheck

#!/bin/bash

for shnfile in *.shn ; do
shorten -i "$shnfile"
done

to append seek tables to all shn files in a directory:

sktappend

#!/bin/bash

for shnfile in *.shn ; do
shorten -k "$shnfile"
done

to remove seek tables from all shn files in a directory:

sktremove

#!/bin/bash

for shnfile in *.shn ; do
shorten -e "$shnfile"
done


Also the 'md5check' script from the etree-scripts is nice because it will check md5s that were generated by a windows system with no problem. I found that checking md5s from windows with md5sum -c xxxxx.md5 doesn't work unless you use something like fromdos or dos2unix to convert it to a *.nix readable format first.

I'm not exactly sure, I think I saw the FLAC thing being wrong on WIKI.ETREE...

But as far as I know, FLAC Frontened pads each track while SHNTOOLs crams all the data forward and pads the very end. Are the bad pad audible, I don't know, but I'd say it would cause some kind of noise, it appears as a silence in the wav form, and might not be quit natural.

T

Note that by changing shn to flac in the batch files, these batch files for shntool should all work for flac. You'd of course want to give the batch file an appropriate name (flaclen.bat, or something).

Also, not sure when this was added, but note that at http://wiki.etree.org/index.php?page=FlacFrontend it now says: "The sector align feature of Flac Frontend now works in v.1.7.1."

But be sure to read the note in the txt file that is installed, FLAC frontend readme.txt:

"Align on sector boundaries:
Align encoding of multiple CD format WAVE files on sector boundaries. This option is only allowed when encoding WAVE files, all of which have a 44.1kHz sample rate and 2 channels. With this option enabled, the encoder will align the resulting .flac streams so that their lengths are even multiples of a CD sector (1/75th of a second, or 588 samples). It does this by carrying over any partial sector at the end of each WAVE file to the next stream. The last stream will be padded to alignment with zeroes. This option will have no effect if the files are already aligned (as is the normally the case with WAVE files ripped from a CD).

WARNING: The ordering of files is important! Make sure that the files are in the right order on the list. Best way to achieve this is to number your files ("01 - title1.flac" "02 - title2.flac" etc.). "

Assuming that does in fact work, one brute force approach to fixing the SBEs is to decode flac files to wav; then re-encode them to flac to fix the SBEs; then decode to wav again to burn to CD. Not the speediest approach, but it may be the easiest for most people. Be sure
- the files sort in track order, because that's the order they are processed in; otherwise you're moving bits of the tracks around to the wrong places; and
- you have a new enough version of flac frontend

Obviously creating new flac files raises issues about another seed, etc., as discussed above; but if you're doing this to just burn the wavs to audio discs with no microgaps - then the brute force approach might be the simplest approach.

Using "align on sector boundaries" is also, of course, a way to fix SBEs when creating new flac seeds for newly tracked shows. Yet another reason to use flac instead of shn for any new seeds.

FWIW here's an example of how flac frontend does sector boundary alignment and how shntool does it.

The net result (at least in this case) seems to be you can do a decent job fixing sector boundary errors with flac frontend. So the brute force approach to fixing SBEs (decode/re-encode/decode/burn); or use of "Align on sector boundaries" with new seeds would seem to work in flac frontend.

FYI, one sector = 588 samples (44,100 samples per second x 1/75th of a second). Each sample uses 4 bytes (16 bits per sample x 2 channels / 8 bits per byte). So a sector is 2352 bytes on a CD audio disc.

original files - shntool len output showing only some tracks with SBEs


flac frontend, using "Align on sector boundaries"

Note that flac reports the number of SAMPLES moved - each sample is 16 bits x 2 channels / 8 bits per byte, or 4 bytes per sample. Also discussed in the next section of this post - shntool uses bytes instead of samples for this info.



shntool fix

If you use the default options on shntool - make all shifts backward (that is, toward the beginning of the disc), and pad the last file with zeros - it generates exactly the same file sizes as flac frontend. Here's the shntool fix preview output for that - which describes the BYTES moved (rather than SAMPLES).

output for the fix preview command: shntool fix -o flac -d temp -noskip -p *.flac



Alternatively, if you don't pad the last file and round to the nearest sector, you get the following results.

output for the fix preview command: shntool fix -o flac -d temp -nopad -noskip -s r -p *.flac



Note that the use of the -nopad switch is the reason why the last file is not padded and still shows an SBE. That's something of a personal preference to allow for:
- The possibility of rejoining files tracked over several discs later on, without remembering to edit out the bit of digital silence
- Expecting that it will get padded with zeros anyway, when burned to disc

Using the round option is aimed at trying to get the new track index points as close as possible to the original track index points.

Not too much time left aorund here for me to really digest all that, but thatnks Buzzy...

Does all that mean the Frontend SBE repairs and the SHNTOOLs SBE repair produce the EXACT same results, in that the WAV MD5 for one is the same as it would be for the other???

Terry

PS edited to remove large "quote"

You know, I didn't test the wav md5s and I've deleted the old files so I can't test it now. Have to save that for next time.

But I'm willing to bet that the short answer is yes. (The bytes got moved the same way, and I'm willing to trust that those bytes are the same samples in each case.)

Of course that assumes you are comparing shntool when it's using comparable options to what's hardwired into flac - the default backward shift, in particular. shntool gives more options, which would give slightly different (and in some cases maybe very slightly better) results.

The practical advantage of shntool, if you're fixing an already encoded set of flacs, is that you can effectively output directly to flac. Fewer steps, less spare hard drive space needed.

The other issue is tags - presumably the tags get lost in either case, as I can't think of any way they'd get passed on automatically. Another issue for another time.

(PS - This was already a long thread - any chance of deleting the long quoted part? It wasn't all that interesting the first time ... )

Some refinements, if you use the batch files a lot -

If you just use the batch files as listed above, you'll need to copy the program (such as shntool) and the batch file to the directory you're using it in - so that the batch file operates on that set of files, and so the batch file can find shntool or shorten. You can save a couple steps if you set up your computer a little ...


Putting shntool and shorten where Windows can find them

You can put a copy of shntool in one of the system folders (one of the ones that's listed in the Path in the Environment Variables), so shntool can be found anytime it's needed. Look in Control Panel, System, Advanced, Environment Variables button at the bottom. For example, C:\Windows\system32\

Or, if you'd rather not start burying stuff in your Windows folders - you can add the directory where shntool or shorten is to the Path list. For example, drop a copy of shntool into a folder called C:\Utilities and add C:\Utilities to the Path list, and any program will be able to find it.


Adding commands to the Windows Context menus (right-click menus)

Depending on how fancy you want to get ... you can add the batch files to the right-click context menus. Once it's set up, it makes it very easy to check shns for seek tables and for errors (using the shntool len command). It may not be precisely these steps on your computer, but it's very close:

Open My Computer (or any folder), Tools, Folder Options, File Types. Find shn. Click Advanced (button at the bottom, under Details for shn extension), then New.
- For "Action:" use: Check shns for seek tables
- For "Application Used ..:" something that points to the batch file you created, like: C:\Utilities\shn-inquire.bat "%1"
- Check the DDE box

The "%1" automatically passes along the file name you click on. Also, this assumes you've got a file called shn-inquire.bat in C:\Utilities , which has something like this inside it to run the commands:

for %%f in (*.shn) do shorten -i "%%f"
pause

Similarly, if you create shn-len.bat you can add another new entry:

Create shn len file
C:\Utilities\shn-len.bat "%1"

Then when you right-click on a .shn file, you should see options to

Check shns for seek tables
Create shn len file

When you select it for one file, it should do the whole folder of files (the way the batch files in the posts above are written).

That's not entirely step by step but hopefully enough to get the idea. Comments and clarifications appreciated if you try this. Or, if there's an easier way, let us know.


By the way, you can do something similar to check flac files for errors:

Create flac len file
C:\Utilities\flac-len.bat "%1"

Which assumes you have a file called flac-len.bat in a folder called C:\Utilities, with something like this inside the batch file:

@echo off
shntool len *.flac > len.txt
start notepad len.txt

if you're on a mac, xACT does this all for you. no command line necessary....

On Windows, Trader's Little Helper Trader's Little Helper is a front end and installer that will do most of this for you, too. What's been implemented so far seems to work fine, though there are still some features in shntool that need a command line or batch file. A big help, though!

So you really ought to be able to at least check every show you seed or share to see if it has SBEs. And if it does, it's worth a little more effort to fix them.

You do still have to be careful to
- make sure the files sort in the right order; and
- consider the effect that any fades added at the end of discs might have. (That is, if the files already have fades at the start or end of the discs, do a disc at a time; if they don't, do the whole show at once.)