Skip to main content
Indiana Wesleyan University Support Knowledge Base

File Management Methods

  1. Last updated
  2. Save as PDF

Overview

Lists available options and methods for managing files using ExportManager.

Details

 

Setup

The following setup & options are available for file management:

FieldDelimiter

String used to separate each field when writing a delimited file using WriteToDisk()
Default:  comma

RecordDelimiter

String used to separate each record/row of data when writing a delimited file using WriteToDisk()
Default:  CRLF

IncludeHeader

Boolean value indicating whether column header labels should be on line 1 of the delimited file
Default:  true

UseQuotes

Boolean value indicating whether data fields should be surrounded by double-quotes
Default:  true

DateFormat

String specifying the .NET Framework custom date format to use when writing out datetime columns
Default:  yyyy-MM-dd

Footer

String of text to append to the end of the delimited file on a new line
Default: empty string (none)

NoArchive

Boolean value indicating whether the noarchive command line argument is in effect
Default:  false

WorkingSet

The WorkingSet object is an instance of the FileManager class which represents the working set of data in the currently running export.  It has the following methods and properties available:

IsFileImported
Boolean value indicating whether files from external sources have been added to OtherFiles in this working set
IsFileWritten
Boolean value indicating whether any DataFiles (from SQL server) have been written to disk
IsDataLoaded
Boolean value indicating whether data has been retrieved from the SQL database
DataFiles
List of FileInfo objects set by AddOutputFilename() to be used for writing SQL data to disk
OtherFiles
List of FileInfo objects pointing to files in the working set that have been imported from external sources with ImportFilesFrom()
DataPath
DirectoryInfo object representing the full path to the root directory of the working set for this export
ArchivePath
DirectoryInfo object representing the full path to the root directory of the archive folder for this export
SelectFiles( int? daysOld, params string[] files )
Take a list of file path strings optionally including wildcards and resolve them to a list of FileInfo objects optionally filtering out files not changed within the past few days.

Execution

The following methods are available for file management.  They are all defined in the FileManager class, but wrapped in the Export class for convenience:

ImportFiles()

(Overloaded) Copy or move files from an external source into the current working set.  Arguments allow for bringing the files in under a subdirectory, deleting the source files, filtering out files older than X days, and wildcard file selection.

/// <summary>
/// Copy files from another location into the working set for this export. Use this to bring in files
/// from an external process that need to be sent out to the export's final destination.
/// </summary>
/// <param name="subdir">Optional subdirectory to create under DataPath for the file(s)</param>
/// <param name="deleteSource">Use True to delete the source file(s) after copying</param>
/// <param name="daysOld">Only import files that were changed within this many days ago</param>
/// <param name="files">One or more files to bring in to the working set</param>
/// <returns>True if one or more files were successfully copied. Otherwise false.</returns>
public bool ImportFiles( string subdir, bool deleteSource, params FileInfo[] files )
public bool ImportFiles( string subdir, bool deleteSource, List<FileInfo> files )
public bool ImportFiles( string subdir, bool deleteSource, int? daysOld, params string[] files )

GetWorkingSet()

/// <summary>
/// Returns a list of all files in the working set regardless of whether they were added using standard functions.
/// Warns if extra files were found.
/// </summary>
/// <returns>List of all files in the data directory (working set) for this export including subdirectories.</returns>
public List<FileInfo> GetWorkingSet()

ExportFilesTo()

/// <summary>
/// Copy the current working set of export files to another location.
/// </summary>
/// <param name="targetPath">Full local or UNC path destination for the file(s)</param>
/// <param name="subDir">Optionally export only files and directories in this subdirectory of the DataPath</param>
/// <returns>True if one ore more files were successfully copied. Otherwise false.</returns>
public bool ExportFilesTo( string targetPath, string subDir = "" )

ArchiveFiles()

Zip up all files in the current working set to an archive in the archive directory for this export, delete the working set files, and expire old archive zips according to arguments passed.  Arguments allow for expiring old archives based on age in days, total number of files, or both.

/// <summary>
/// Archive the current working set of files and expire any old archives
/// </summary>
/// <param name="daysToKeep">Number of days back to keep old archives. Files beyond this are deleted first. Use 0 for no expiration.</param>
/// <param name="filesToKeep">Max number of archived files to keep. Oldest files are removed until this number remain. "daysToKeep" expiration happens first. Use 0 for no maximum.</param>
public void ArchiveFiles( int daysToKeep, int filesToKeep )

ClearFiles()

Delete all files and directories in the current working set.  The files are not archived or otherwise saved first.

WriteToDisk()

(Overloaded) This method takes the ExportData.Tables datasets and writes them all to disk provided there is a corresponding output file that was added using AddOutputFilename().  You may also specify an integer argument to tell the method to only write a single dataset to disk with the given index.  If you tell it to writeHash, it will also compute a sha256 hash for your file and automatically compare it to the remote after SFTP upload (if the server supports it).

/// <summary>
/// Writes the SQL data in all result sets to disk in delimited file format
/// </summary>
/// <param name="writeHash">In addition to the file, also write the SHA256 hash</param>
/// <param name="fileName">Add this file name to the output list and write the SQL data from its result set to disk in delimited file format.</param>
/// <param name="tableIndex">Write out only the specified index of the result sets and matching DataFiles</param>
public virtual void WriteToDisk( bool writeHash = false )
public virtual void WriteToDisk( string fileName, bool writeHash = false  )
public virtual void WriteToDisk( int tableIndex, bool writeHash = false  )

Example

Script

The following example copies some images into the working set, writes SQL results to disk (for full SQL example see SQL Querying Methods), exports them to a remote location, and archives them.  It then checks if the noarchive command line argument was specified and if so, manually deletes everything in the working set, clearing it up for the next run.

public FileSample( string[] args ) : base( args )
{
    // Delimited file options
    IncludeHeader  = true;
    UseQuotes       = false;
    FieldDelimiter  = "|";
    RecordDelimiter = "\r\n*****\r\n";
    DateFormat      = "MM-dd-yyyy";
    Footer          = "----------------------------------------------\r\nCopyright (c) 2016 Indiana Wesleyan University";
}

public override void Run()
{
    // Import some files into a subdir in the working set named extras - wildcards supported
    ImportFiles( "extras", false, null, @"C:\Users\Public\Pictures\Sample Pictures\*s.jpg" );
    
    // Assuming SQL data was selected, write it to delimited files using settings
    // in the constructor
    WriteToDisk();

    // Copy all working set files out to another location
    ExportFilesTo( @"N:\SampleExportTest" );

    // Zip up files and keep the past 30 days in the archive directory
    ArchiveFiles( daysToKeep: 30, filesToKeep: 0 );

    // If the noarchive command line option was specified, manually delete out the 
    // working set of files
    if ( NoArchive )
        ClearFiles();
}

Log output

Expected ExportManager.log output:

2016-09-14 11:12:38.3088 INFO -------------------------------------------------------------------------------
2016-09-14 11:12:38.4148 INFO IWU Export Manager launched
2016-09-14 11:12:38.4188 INFO -------------------------------------------------------------------------------
2016-09-14 11:12:38.4188 TRACE OS=Microsoft Windows NT 6.1.7601 Service Pack 1  Cores=4  64bit=True
    Machine=IWU71563  Domain=IWUNET  User=konrad.willmert
    Command=ExportManager  FileSample
-------------------------------------------------------------------------------
2016-09-14 11:12:38.4458 DEBUG Export "BasicSample" found
2016-09-14 11:12:38.4558 DEBUG Export "ColleagueSample" found
2016-09-14 11:12:38.4558 DEBUG Export "ColleagueTest" found
2016-09-14 11:12:38.4718 DEBUG Export "FileSample" found
2016-09-14 11:12:38.4798 DEBUG Export "SampleExport" found
2016-09-14 11:12:39.3348 INFO Executing "FileSample"
2016-09-14 11:12:40.1548 DEBUG "FileSample" export process executed in 1,673ms
2016-09-14 11:12:40.1658 DEBUG IWU.ExportManager.Exports.FileSample FileManager@C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\data\FileSample: 0 data files, 0 other files ColleagueConnection mocha.task@DEV with database "coll18_dev" and HOLD dir "\\ERPapp\d$\datatel\ftp\coll18_dev\data\_HOLD_"
2016-09-14 11:12:40.1658 DEBUG ExportManager total run time: 1,860ms
2016-09-14 11:12:40.1658 INFO Exiting.

Expected FileSample.log output:

2016-09-14 11:12:39.3558 INFO -------------------------------------------------------------------------------
2016-09-14 11:12:39.3558 INFO IWU.ExportManager.Exports.FileSample initializing.
2016-09-14 11:12:39.3558 INFO -------------------------------------------------------------------------------
2016-09-14 11:12:39.8478 WARN TEST MODE is enabled. No actual data files will be exported/uploaded. Instead, write access will be tested. Override with "notest".
2016-09-14 11:12:39.8598 DEBUG Data directory: C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\data\FileSample
2016-09-14 11:12:39.8598 DEBUG Archive directory: C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\archive\FileSample
2016-09-14 11:12:39.8698 INFO Importing file(s) into working set.
2016-09-14 11:12:39.8698 DEBUG Created directory: C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\data\FileSample\extras
2016-09-14 11:12:39.8858 DEBUG Copied "C:\Users\Public\Pictures\Sample Pictures\Hydrangeas.jpg" to "C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\data\FileSample\extras\Hydrangeas.jpg"
2016-09-14 11:12:39.8858 DEBUG Copied "C:\Users\Public\Pictures\Sample Pictures\Penguins.jpg" to "C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\data\FileSample\extras\Penguins.jpg"
2016-09-14 11:12:39.9068 DEBUG Copied "C:\Users\Public\Pictures\Sample Pictures\Tulips.jpg" to "C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\data\FileSample\extras\Tulips.jpg"
2016-09-14 11:12:39.9068 INFO Successfully copied 3 file(s).
2016-09-14 11:12:39.9068 DEBUG Successfully copied 1,994,007 bytes in 3 file(s) in 37ms
2016-09-14 11:12:39.9188 INFO Exporting working set to: N:\SampleExportTest
2016-09-14 11:12:39.9188 DEBUG Created directory: N:\SampleExportTest
2016-09-14 11:12:39.9408 DEBUG Test file copied successfully.
2016-09-14 11:12:39.9678 INFO Archiving working set.
2016-09-14 11:12:40.1468 DEBUG Archive created: C:\Users\konrad.willmert\Source\Repos\IWUExportManager\IWUExportManager\bin\Release\archive\FileSample\FileSample 2016-09-14 11.12.39.zip
2016-09-14 11:12:40.1548 DEBUG Working set deleted.
2016-09-14 11:12:40.1548 INFO Expiring old archives.

Considerations

  1. Back to top
  • Was this article helpful?

Recommended articles

  1. Article type
    Reference
    Stage
    Final
  2. Tags
    This page has no tags.