Articles in this section

Bulk Export

Jann DiPaolo
  • Updated
Follow

Bulk Export can be used to export all or part of the FYI document store, including key metadata, to a designated Amazon Web Services (AWS) S3 directory. It can be used to export documents that are held in a specific Cabinet or to export documents for a specific Partner or Client.

Bulk Export could be used in the event of a practice split. Bulk Export can also be used to export the documents for a specific client which can be used if a client has left your practice and you want to export all their documents before deleting them, or to archive documents, such as for a yearly archive.

Exporting all your documents would be used in the rare occasion that your practice chooses to leave FYI and it allows you to export documents in a well-structured fashion to then adopt a new system.

There is also a function to export to your practice OneDrive to the folder FYI - Export. Refer to Bulk Export OneDrive.

Important note: Bulk Export is only available on demand.
To run a Bulk Export, you need to contact the FYI Support Team at support@fyidocs.com to enable this function.

You can only request one export at a time.

This article contains the following sections:

Index .csv File and Additional Metadata

As part of each Bulk Export, a report.csv file is automatically created with a summary of the files that were exported. This includes the following columns:

Document Name
File Path
Modified FileName (the filename as exported)
Document Type
Category
Reference Number
Created On
Created By
Last Modified On
Last Modified By

Note: 

  • If you have included any additional columns in the export using the Configuration - Structure List, these columns are also included in the report.csv index file of the export (refer below to Step 1 - Set up the Configuration Structure for the Exported files).
  • If re-importing files to a new FYI, ensure that all report.csv files are included in the folder location 'export-data' for Migrate App. This will allow this metadata to be applied to the imported documents.

The Index .csv file is created at the end of the process and added to the root directory of the export, 'export-data'.

If there is any additional metadata that you want to hold as a record as part of the export (such as Partner, Manager, Tags, Keywords) you can separately export a list from an FYI documents list as a record.
When you run an export, you can checkmark Include FYI Reference so that all the documents being exported have the FYI Reference Number appended to the end of the document name. By exporting a documents list in FYI with the Reference Number as the leading column, and including the columns for any additional metadata you want to keep as a record, the FYI Reference Number can be used to cross-reference to the exported documents list (refer to Exporting Lists). Note that you can export up to 10,000 rows so you may need to export more than one list.

Notes on Bulk Export

  • Only the current version of a document is exported. The Bulk Export does not export any prior versions of the document.
  • The Export is only available to FYI Admins.
  • You can only have one instance of the Export running at one time.
  • Archived documents will be included in the export.
  • Bulk Export exports Draft Emails.
  • Bulk Export exports Web Links. The file extension of the exported files is .dat
  • Bulk Export exports Phone Calls, File Notes and Meetings. The file extension of the exported files is .html.
  • If the category contains a backslash (\) the export converts this to an underscore (_). For example, if the category name is "GST\BAS" this is converted to "GST_BAS".
  • We recommend planning your export around our weekly software release on a Thursday evening. A software release restarts the export process and can extend the time for the process to finish.

Security

Public and private keys are generated for you by FYI. Access to the exported files is secured by the key pair.

  • The public key is held for you by FYI.
  • The private key is linked to a username, which is your email address.
    The private key is generated as part of the Export as a .pem file. It is downloaded directly as part of running the Export in FYI.

Note: You must keep the username and the private key secure at all times as this provides access to your documents.

The Export security is controlled by an SFTP client (SSH File Transfer Protocol/Secure File Transfer Protocol). We recommend using Cyberduck for this. Another SFTP client can be used if you are familiar with SFTP clients or have one installed already. If you need more information on SFTP clients, refer to a reference source such as this Wikipedia article SSH File Transfer Protocol.

Steps when Running the Export

To run a Bulk Export, make sure you have contacted the FYI Support Team at support@fyidocs.com to enable this function. To run the export you will need to do the following:

Step 1 - Set up the Configuration Structure for the Exported files

The first step is to configure the structure of the folders that will be used for the exported files.

  1. Log in to FYI as an FYI Admin.
  2. From FYI, click Settings in the top right-hand corner.
  3. Select Practice settings from the drop-down, then select Documents.
  4. Go to the Import/Export tab.
    This displays a log of the import and export functions that have been run or that are currently in progress.
  5. Click the Configure button.
    1561_Configure.gif
    The Import/Export setup displays.

  6. Go to the Export tab.

    1161_Export_Structure.gif

The Structure List determines the hierarchy that will be created by the Export. This determines the top-level folder and any sub-folders for the Export.

For example:
If the structure is set up as Client Name / Year / Cabinet, a top-level folder is created for the Clients and the documents will be exported to a folder for the relevant Cabinet within Year.

If the structure is set up as Group Name / Client Name / Year / Cabinet, a top-level folder is created for the Client Groups and the documents will be exported to a folder for the relevant Cabinet within Year within Client.

You can remove any metadata groupings from the Structure List and add any to it from the Available list. You can also change the sequence of the metadata in the Structure List.

  1. Click to select what you want to move.
  2. Drag and drop it either to the Structure List to include it, or drag it to the Available List to remove it from the Structure List. 
  3. Drag it up or down in the Structure List to change the sequence.
  4. Click Save.

Note: You can check the Structure List and also make any changes when the export is run. Refer below to Step 3 - Running the Export.

Note: If documents have already been exported and you needed to change the structure, this will not affect the folders in the previous export. Every export is saved to a separate folder and uses the structure as set when the export is run. The folder name of each export is set as the date and time when the export request was started.

   Return to Steps when Running the Export

 

Step 2 - As relevant, move Documents to be exported to a specific Cabinet, and/or check the user assigned as the Partner for the Documents

If you want to export documents for a specific Partner or Client, you have several options.

You can export documents for a specific Partner. This exports documents for clients where the selected user is held as the Partner for the Client. Include Partner as a column in the Clients list and in the Documents list to check the documents that will be exported if you are going to export using the selection for Partner.

You can also select a specific cabinet or all cabinets. If you are not exporting all documents, or not exporting for a specific Partner, ensure that the relevant documents are in a specific cabinet. You can move any documents using Bulk Update (refer to Bulk Updating of Filing Details).

To export documents for specific clients it's easier to run a full export, and then use your SFTP Client (for example, Cyberduck) to select the particular clients you require. Hold the Ctrl button to select multiple clients; refer to the instructions below on using an SFTP client.

   Return to Steps when Running the Export

Step 3 - Run the Export

When running the Export the first time, you download the private key, the .pem file.

    1. From FYI, click Settings in the top right-hand corner.
    2. Select Practice settings from the drop-down, then select Documents.
    3. Go to the Export tab.
    4. Click Run.
      The Export displays.
    5. Ensure the Medium is set as "AWS Transfer"
      1564_Export_AWS.gif

    6. As relevant, select the Partner, Client or Cabinet. Or select a combination of these.
      When exporting documents for a Partner this exports documents for clients where the selected user is held as the Partner for the Client.
      Selecting a Client exports the documents for that Client.
      You can also select the Cabinet that holds the documents you want to export. To export all Cabinets leave this as "No option selected". This can be used instead of or in combination with Partner or Client.

    7. You can run the Export as Incremental so that it exports only documents since the export was last run, not the whole document system.

    8. When Include FYI Reference is check-marked, all the documents that are exported have the FYI Reference Number appended to the end of the document name.
      Refer above to Index .csv File and Additional Metadata about the index .csv file that is automatically created for each export. This includes the FYI Reference Number.
      If you need to keep a record of any additional metadata you want to keep as a record you can include the FYI Reference Number in the exported documents to cross-reference to the information in any additional exported documents list.

    9. Click Generate.
      This generates a .pem file with private keys.
      1563_Export_AWS_download_new_key.gif

    10. Click the link Your .pem file is ready. Click here to download it.

    11. Save the downloaded .pem file somewhere safe.

    12. Checkmark "I have downloaded the .pem file" to confirm you have done this.

    13. Click Next.
      The Export (step 2 / 2) displays. This shows the structure that has been set up. Check this is correct for the export you want to run.
      You can make any changes to apply to the export as required (refer above to Step 1 - Set up the Configuration Structure for the Exported files).

    14. Click Create to start the Export.

      The Import/Export tab re-displays and shows the Export request.
      When the Export is in progress, a grey progress bar displays below the Export request on the Import/Export tab

      Note: For a few seconds, the Cancel button displays which can be used to cancel the Export at this point. Once the Export request is picked up by FYI it will run until finished.

For any long filenames, the filename of an exported file will be truncated to 100 characters.

   Return to Steps when Running the Export

Step 4 - Download and Install the SFTP Client (Cyberduck)

Downloading and installing the SFTP Client is a one-off process. We recommend using Cyberduck and the examples below are shown using this as the SFTP to access the exported documents.

Click the link below to download and install Cyberduck.
https://cyberduck.io/download/

   Return to Steps when Running the Export

Step 5 - Run the SFTP Client (Cyberduck)

When you start the Cyberduck application, the following screen displays.

1557_Cyberduck_1.gif

  1. Click Open Connection.
  2. Choose “SFTP (SSH File Transfer Protocol“ from the drop-down.

    1558_Cyberduck_2.gif

  3. Fill in the following fields:

    Server  - Enter the server as:
    s-87445f28b4fb451ea.server.transfer.ap-southeast-2.amazonaws.com

    Username - Enter the full email that you use to log in to FYI.
    SSH Private Key - Click Choose and select the .pem file provided to you by FYI. This file holds the private key.
    Password - Leave the Password field blank.

    Port - Do not change the Port number.

  4. Click Connect.

  5. If an Unknown Fingerprint pop-displays, click Allow.

    1559_Cyberduck.gif

  6. When you log into the SFTP, you see a folder for the export with the folder name set as the date and time when the export request was started. Below this are the folders for the export structure that you specified.

   Return to Steps when Running the Export

Step 6 - Extract the Files from Cyberduck

The next step is to extract the files from Cyberduck.

The top-level folder will be displayed in the main viewing screen of Cyberduck. You can extract files from the top-level folder, or choose a sub-folder within it to display the folder you want to extract from.

Note: The Index .csv file is added to the root directory of the export, in the folder 'export-data'.

If you want to calculate the size of the export in AWS before extracting:

  1. In Cyberduck, right-click over the folder you want to extract from.
  2. Click Info.
     The following menu displays.
    2470_Bulk_Export_Calculate_Size_of_Export.gif
  3. Click Calculate and let it run through all files to calculate the size of the export.
    This may take some time, depending on how many files are hosted.

To extract the files:

  1. Right-click over the folder you want to extract from and select Download To.
    Tip: To select multiple folders hold the Ctrl button and click on each of the required folders
  2. In the Browse For Folder, select the folder you want to save the files to.
  3. Click OK and the files download to that location.

This leaves the data in Cyberduck after extraction, so they can be extracted again if required.

Note: If you choose OneDrive as the location, the download may be slow and it may impact FYI syncing. It is recommended to first save to your local hard drive. From there, you can then save the files to your OneDrive, for example overnight, or you can copy them to a portable USB or hard drive as required.

   Return to Steps when Running the Export

Running the Export Again

For any subsequent runs of the Export, you do not need to generate a private key again. The previously generated private key gives you access to the newly exported documents.

If you need to change the Export structure, this can be done from the Export Configuration (refer above to Step 1 - Set up the Configuration Structure for the Exported files) or can be changed when running the Export.

Every Export is saved to a separate folder with the folder name set as the date and time when the export request was started. If you re-export the same documents this will not overwrite any previous exports.

  1. On the Export screen, make your selection for Medium.
  2. If relevant, make a selection for Cabinet.
  3. Click Next.

The Export structure is displayed, which can be changed if needed and you continue as above.

Note: If the keys need to be regenerated, users will need to run the export again starting at Step 1.

Accessing the STFP as a Different User

Only one user is authorised to access the SFTP at a time.

The .pem file is linked to the username which is the email of the person who originally raised the request.

If another user needs to access the SFTP temporarily, that person would need the .pem file and would need to use the original requestor's email as the username. They do not need the original requestor's password.

If another user needs permanent access, they can regenerate the private key as above with their username. After regenerating, the new user will have access to previous exports as well as any new exports. The original user will not have access to previous or new exports.

Re-Importing Exported Documents to FYI

If documents have been exported with this Bulk Export and need to be re-imported to FYI, they can be imported with the FYI Migrate app and selecting the Windows Explorer option. Refer to Using the FYI Migrate App to Sync your Data and Document Migration for Windows Explorer, HowNow or HandiSoft.

Note: Ensure that all report.csv files are included in the folder location 'export-data' for Migrate App. This will allow the metadata to be applied to the imported documents.

 

Related articles