Smaller file size, lower data rate, less computational overhead, no conversion loss.
A 64 bit float requires 64 bits to store.
ASCII representation of a 64 bit float (in the example above) is 21 characters or 168 bits.
Also, if every record is the same then there is a huge overhead for storing the name of each value. Plus the extra spaces, commas and braces.
So, you are at least doubling the file size and data throughput. And there is precision loss when converting float-string-float. Plus the computational overhead of doing those conversions.
Something like sqlite is lightweight, fast and will store the native data types.
It is widely supported, and allows for easy querying of the data.
Also makes it easy for 3rd party programs to interact with the data.
If you are ever thinking of implementing some sort of data storage in files, consider sqlite first.
Never said it had to be a text file. There are many binary serialization formats that could be used. But is a lot of situations the overhead you save is not worth the debugging effort of working with binary data. For something like this that is likely not going to be more then a GB or so, probably much less it really does not matter that much if you use binary or text formats. This is an export format that will likely just have one batch processing layer on. This type of thing is generally easiest for more people to work with in a plain text format. If you really need efficient querying of the data then it is trivial and quick to load it into a DB of your choice rather then being stuck with sqlite.
As long as it is something I want to be loading all at once, I would go with JSON and use the JSON binary format provided by QJsonDocument and forget about it.
But if I expect it to get bigger and need stuff partially loaded, things get lengthier.
Stream Reading libraries (e.g. the XML one in Qt) makes it much longer to write and that’s where I would consider whether to use a database instead.
Smaller file size, lower data rate, less computational overhead, no conversion loss.
A 64 bit float requires 64 bits to store.
ASCII representation of a 64 bit float (in the example above) is 21 characters or 168 bits.
Also, if every record is the same then there is a huge overhead for storing the name of each value. Plus the extra spaces, commas and braces.
So, you are at least doubling the file size and data throughput. And there is precision loss when converting float-string-float. Plus the computational overhead of doing those conversions.
Something like sqlite is lightweight, fast and will store the native data types.
It is widely supported, and allows for easy querying of the data.
Also makes it easy for 3rd party programs to interact with the data.
If you are ever thinking of implementing some sort of data storage in files, consider sqlite first.
Never said it had to be a text file. There are many binary serialization formats that could be used. But is a lot of situations the overhead you save is not worth the debugging effort of working with binary data. For something like this that is likely not going to be more then a GB or so, probably much less it really does not matter that much if you use binary or text formats. This is an export format that will likely just have one batch processing layer on. This type of thing is generally easiest for more people to work with in a plain text format. If you really need efficient querying of the data then it is trivial and quick to load it into a DB of your choice rather then being stuck with sqlite.
As long as it is something I want to be loading all at once, I would go with JSON and use the JSON binary format provided by
QJsonDocumentand forget about it.But if I expect it to get bigger and need stuff partially loaded, things get lengthier.
Stream Reading libraries (e.g. the XML one in Qt) makes it much longer to write and that’s where I would consider whether to use a database instead.