CSV usage - collection manager tasks (newbie)

If you have read the articles about CSV files - an introduction and CSV - stages of evolution, then you now know that csv files are used to index your files into reasonable chronological chunks, most often DVD chunks.

This article is about which tasks there is in organizing your files so that they match those csvs. To utilize csv files you must use a program. Much in the same way you would a BitTorrent Client and a .torrent file. The programs in question are Collection Manager tools. This article is not about a specific collection manager program, but instead of the general tasks within these programs.´

Directories:
There are a minimum of 4 directories involved with a collection manager. The first one is the directory where you will keep all your csv files. This directory is referred to as your csv folder. The second directory involved is the directory for your reports. Reports are text files which hold the result between your multimedia files and your csv files. The csv folder and the report folder need to be 2 different places. The 3rd directory involved is your collection folder. The collection folder is the place where you keep all your CD's or DVD's, this is the folder where your Hegre.-Art-DVD41 folder and data will reside. The last folder is your download folder. This is where you dump all your data before they get organized. Stuff from usenet, websites and every other place where you get data from.

1. CSV folder
2. Report folder
3. Collection folder
4. Download folder

You will need to configure your collection manager for these 4 directories as a minimum. You can have multiple collection folders aswell as multiple download folders. Start off by copying your csv files to the csv folder. Empty the folder for the reports, copy what you do have into the collection folder in the proper sub folders. Do a little cleanup in your download folder.

Setting up the first collection:
The next thing is to find a csv for some of the stuff you do have and know should match some files. Let's pretend that you know you have some files which supposedly should belong to Hegre-Art-DVD41. So find the Hegre-Art-DVD41 csv on a csv site and download it. Use the collection manager to setup a new collection with the csv you have downloaded. And point the collection towards the directory where you have stored the files from Hegre-Art-DVD41.

Check:
Depending on the tools of choice then the collection manager will check the files you have in the directory, against the files specified in the csv. The csv for HGRARDVD41 (the trigger name for the csv in question) has 1673 files registered. After the collection manager has checked the files on your harddrive, then it will have show you how many files you have complete, compared to the csv. If everything is there and match 100% then you get a count of 1673 and a missing/wrong count of 0.

If you get a missing/wrong count which is above 0, then some files are missing or not the original files which matches the csv. This means that even though you may be able to view the picture or watch the video, then someone has altered something in the file. Theas files will me moved to a directory outside the collections directory. If you are lucky then the files which does not match the csv you are checking, will match to some other csv, let's say HGRARDVD40 as example. The directory where the bad files reside in is a good source as a 'hunt' directory.

Hunting:
Hunting is the term used to scan and look for files which match a csv. Often you would hunt from your download folder, which possibly contains alot of ramdom files, jpg, avi, mov, rar, zip etc. The files which are discarded by a check may match other csvs. You will have to empty you "bad files" directory from time to time, unless you have unlimited space and wants to keep stuff which does not match anywhere. Modern Collection Management tools can look inside zip and rar files. So there is no need to extract the files yourself, unless there is some damage to the archive. You can configure the Collection Management to hunt from any number of files residing on multiple sources. When hunting the Collection Management tools are able to delete a found file from the hunt directory if it matches a csv, thus cleaning your hunt folder while hunting. This ability also goes for the files inside zip and rar archives.

Reports:
Everytime you check or hunt then the Collection Management tool will produce a report. This text file will show detailed information about which files are good and which files are bad. Also information about sizes can be found in the report. The report can be used in various ways, both by the tool and you. The reports are especially useful when you have collections on offline storage.

CSV (Final) - stages of evolution

This is a followup to the article "CSV files - an introduction"

CSV files or ecsv files to be specific follow specific steps of evolution. These steps are to complete the csv file so that it ultimately holds a number of files which consume the maximum amount of space on a storing media as a CD or DVD etc. Furthermore the stages have the goal of keeping some kind of chronological order.

Ongoing
The stages are set by the csv maintainer, more about that role in another article. The maintainer starts off by having an ongoing csv. Sometimes this is explicitly  shown when a part of the file name being (ongoing). Most often there is no explicit naming convention to an ongoing csv. This is the case with ongoing csvs from TVG (The Vixen Grimoire) and the former ET (El-Toro).

Pre-Final
When the maintainer has collected enough files to fill a CD or DVD then the maintainer sets the csv to the stage (Pre-Final). This is to signal people using the ongoing csv that it is almost complete and they should expect that a new ongoing csv will soon emerge. According to community rules the maintainer must keep the csv as Pre-Final for a short while to allow other to check the csv and report errors. If there is an error in the csv, ie. wrong set or wrong naming or missing sets then then maintainer will fix the problem (hopefully) and create a new (Pre-Final-1) csv. This can go on for a long time, but it is seldom seen that Pre-Final's go beyond (Pre-Final-4). Once a csv reaches the Pre-Final stage it will NEVER go back as ongoing. This is breaking the rules, and big no-no.

Final
If the maintainer has not had any reports of errors in the csv and feels that the csv is ready to be burned to a CD or DVD then the maitainer will start the next stage, which is (Final). The (Final) means that it is now safe to burn the contents of the csv to a DVD and the user should not expect any further changes to the csv. If the world was perfect then nothing more happends. Everyone using the csv, burn it to CD or DVD and is able to clear up some space on those HDD's.

Re-Burn
But this is not always the case. Sometimes the maintainer is too fast to go from (Pre-Final) to (Final), sometimes people notice errors too late. This is where things get fuzzy, so sharpen your senses. If the change to the (Final) csv is structural only, ie. new layout or new naming or some corrupt picture was removed for good then the maintainer can issue a (Re-Burn). A (Re-Burn) means that the contents of the previous csv is just to be burned to a new CD or DVD. People having the previous version has not lost anything. (Re-Burn)'s can also go into stages like (Re-Burn-1) and so forth.

Final-X
If the change after the (Final) stage is of non structural nature, then maintainer has to issue a (Final-1) and provided there are more changes later it goes to (Final-2) etc. It is seldom seen that anything go beyond (Final-4).

In rare cases the maintainer will choose to use (Re-Burn) as a final exit for the csv if it has reached (Final-4), because it is flat out embarrassing that the maintainer cannot manage make a proper. This last exit is seen, but it is not the way to do it.

There is no stage beyond Re-Burn-10. Pre-Finals never become ongoing. Final's never go back to Pre-Final stages and Re-Burn's never go back to Final's.

It is a common misunderstanding that the use of Final-1, Final-2 etc is whenever the website in question revamps or redesigns itself. This is NOT the purpose of the Final stages. If a website redesigns and introduces all new resolutions of old contents HD images and HD movies then the new revamped site will most likely get a csv set entirely. See with CrushPhoto and CrushPhoto-New as example.

CSV files - an introduction

Recently I was brought to my attention that a simple entry introduction to CSV files in our context was lacking. This article will try to provide you with an introduction to what it is all about.

Facts

• A csv file is a ASCII text file you can read in NotePad or similar
• The extension for csv files is .csv
• CSV means: comma seperated values
• The format used on Vixen Grimoire, El-Toro, WSC etc is not csv, but actually ecsv
• ECSV means extended csv
• The purpose of an ecsv file is to be able to organize your files
• You organize you files according to an ecsv file with a program
• Programs for organizing is called Collection Managers (click the link for an intro to those)
• ECSV files have 1 strict format
• ECSV files has the file extension .csv
• ECSV files are commonly referred to as just csv files
• ECSV files are maintained updated by 1 person at the time
• It is not allowed to tamper or change the ecsv file made by someone else unless there has been given permission to do so. If you get permission, then YOU become the maintainer.

 A csv file (common name) has 1 file listed on every line in the file. Each file has some values to it. The values are in the following order (inside the csv file)

FILENAME, FILESIZE, CRC32 HEX VALUE, PATH, OPTIONAL COMMENT

An example could be:

img_0750.jpg,438648,A2DF81D5,\2008-11\2008-11-27 - Kelly M\,

The line above tell the following. There is a file, which should have the file name img_0750.jpg. This file has the file size of 438648 bytes and the calculated CRC32 checksum value must be A2DF81D5 (hexadecimal presentation) for the file to be the right file. The file should be located in a folder structure which must be \2008-11\2008-11-28 - Kelly M\. There is no comment to this file.

This means that there is only 1 file which actually matches to be the correct file for that line in that ecsv file. In general the uniqueness is assured by the use of the CRC32 checksum calculation  which is almost perfect for distinguising files from each other.

If you have a picture and you calculated the CRC32 value for the picture it may turn up to be (as example) 52AC2E11. If you then change just 1 pixel and save the file again then the file will NOT have the CRC32 value of 52AC2E11. So the two files do NOT match. Same goes if you change the resolution or scale the file.

The files listed inside the ecsv files on El-Toro, Vixen Grimoire and such places have the CRC32 value of the original files from the websites. NOTHING has been changed, not even a meta EXIF piece of information.

Every csv file itself has a CRC32 value, so it is not possible to change an existing csv file and then claim it is original if the CRC32 values of your file does not match the CRC32 value of the file on the csv sites. Here is an example:

The csv file has the name ErroticaArchives-DVD35(Pre-Final)_1872.csv. 1872 lines with file descriptions is listed inside the csv file, of which 88% is pictures. If you add up the file sizes of all 1872 lines the it amounts to 4,439,771,784 bytes, which is ~94% of a single sided DVD. The csv itself has the CRC32 value of 017C2744 to be original. The trigger name is ERRARDVD35 and it belongs to the group Fine Art Erotica, with sub grouping of Errotica Archives. The type for the csv file is regular (there are other types, like asian, wsc and such).

To use a csv file, you must have a program. Such programs are called Collection Managers. The earliest useful collection manager is ScanSort, followed by Hunter and PicCheck. The current tools of choice are PServeCheck or PSProVerify. These two latter tools can be hard to come by, but know a friend who knows somebody, who's connected and may know where to ask for directions OR ask on your favorite csv site *hint*.

Does Earl fit into a DVD?

Earl Miller, one of the collections which dates back in time quite a bit had not moved from CD into DVD.

Further more EMPDVD has gotten a new maintainer Newfy.

The first DVD in the serie will be DVD5. Everything prior to that still resides on the CD csvs.

Thanks to Newfy.

Getting the proper tease

One of the issues with the ONTE site is the hassel with downloading the proper higest quality version of the picture sets. Some sets come in plain size, others in both plain and large sizes and some, luckily most these days, comes in ultra high quality too.

This article will give you one way to filter out all the unwanted versions of the zipped picture sets.

In order for this to work you have to go to every month you wish to download.

Then you should save every month on the site to a htm file, no need to save the graphics along, just keep it to html only. Secondly you have to remove all newlines, carrige returns and tabs from the htm files..

You can accomplish that by running a TR command on the commandline in the OS of your choice.

The TR should look like the one below.

tr -d "\n\r\t" < onteXYZ.htm > onteXYZ.html

Do that on every page you saved. Once they are all saved to their .html counterpart, then run the command below.

grep -Po "http://[\w.]+/members/(.....)?zips/[-\w]+\.zip(?=..[\w ]+.?\([\d]+.[\d]+ MB\)./div../div.)" onte*.html

That will extract all the picture zip files in the higest quality available. No need to filter stuff manually. And no need to clean up after the download.

What the grep actually does is to look for any http:// link which links to a zip file and have some trailings which consists of two ending div's.

Have a look below to better understand the result.

Click to enlarge

Tease and please for show

The csv for the Only Tease site has had a rest of updates since January 2008. The previous maintainer is MIA and that put a stop to a very good csv set.

BUT!, times change and I can happily annouce that new updates are on the way. A source has sprung and this means that new csvs for the ONTE site is arriving at VG and mirrors.

So that you can all enjoy some nice sorting of your downloads.

Here's some cover teasers.

bangbros to directories

This summary is not available. Please click here to view the post.