CSV file data export time

CSV file data export time
none 0.0 0
  • HMI Model: CMT3072
  • EasyBuilder Pro Version: 6.10
  • I have this HMI setup as a data acquisition system for 6 analog inputs. Data is sampled every second and sometimes the test can run for up to three days, the CSV file that is generated can be up to approximately 20-25 megabytes maybe more. I use the data backup feature to transfer the files to a USB drive. My issue is that it can take up to 30 minutes or more to transfer the data/files to the USB drive. I know it’s a lot of data points but the transfer rate seems really slow. Is there anything that I can do to speed it up?

Thank you

Hi @leadfoote,

Thank you for contributing to our forum. Could you please provide some screenshots of your project so we can get a better understanding of your setup, particularly how the Backup Data and Data Sampling objects are being used?

Thank you!

Its set to store/transfer as many files as possible to preserve data just in case. Its also setup to be able to clear the backlog which is done regularly. So theres usually only one or two files that are transfered at a time.

I tried attaching screenshots but it says i can imbed media in a post.

Hi @leadfoote,

A update has been made, you should be able to add your screen shots now.

Got them uploaded, thank you!

Hi @leadfoote,

One thing we can try is splitting the file during backup. This is done by navigating to the Output tab of the Backup (Global) object. Then, within the Split by section, select Customized file.
Note: The File Source must be a “Historical data sampling object.”

Furthermore, within the Backup (Global) object, could you try exporting the file as a DB file and see if the DB file format is quicker?

Aaron, thank you for your suggestions. I will try them out and run some test to see what we get. Out of curiosity, do you know what the data transfer rate/speed is when going from HMI internal memory to USB?

Hi @leadfoote,

The read and write speed likely depends on system load, but it may be visible within the “System metrics” of your HMI’s settings menu:

Is it possible to save data samples directly to USB in CSV format instead of the HMI flash memory? Also, I read that frequent writing to the HMI memory can accelerate the memory degradation. Six float inputs are logged at one second intervals for up to three days at a time. How long can the memory last at that rate? Do you have any suggestion for better equipment or file/data transfer methods that would be better in this application?

  1. To directly save from the data sampling object to a USB, you can select “Save to USB disk 1” within the Data Sampling object. However, it will save the data as a .db file instead of a .csv file. Therefore, using the Backup object is the best way to export a .csv file. If you want to streamline this process even more, I would recommend setting up some timers to automate the backup process. Below are a few videos on how to set this up:

    If you wanted to email/FTP a CSV instead
    Scheduler Set up
  2. We don’t have a specific estimate as to how long the flash memory will last, as it will depend on the HMI design. Please reference section 3.2.1 of this manual for recommendations related to project design: Link.
  3. For better file/data transfer, we recommend using a SQL database to store all of your data. This will allow you to store all your data directly within a database but does require external setup. Here is a link on how to set up MS SQL: Link