2486 shaares
83 private links
83 private links
karlicoss of the data liberation project HPI explains how to best store and access data moved from various points in the cloud/web/internet to your drives and why databases might not always be the best choice.
TLDR.
Save your grabbed data without any manipulation.
Let the manipulation happen every time you access/interpret the data.
If you have slices of data (mostly time frames), don't try to merge them on disk but save as extra files and merge on access/interpretation as well.
You can make use of databases for access caching since the last points generate some overhead for each access.