Efficient File Storage
Just because you can create container fields doesn't mean you should create them whenever you think you need to store files for a specific table. There are possible solutions which, when implemented creatively, allow you to use one single container field for all of your solution's file storage. True, you may need container fields for other uses, but, when it comes to linking and storing the documents managed within your solution you can get away with one single container field within a single Files table.
In this video, I go over a super simple strategy for managing all of your solution's files within a very efficient storage model. We cover how to identify duplicates and simply prevent them from entering the system. Yet, any given file can be linked to any other record within the whole system.
Beyond this, you gain a very big advantage of being able to use the same scripts and layout elements across the whole solution. By adding a single relationship to any table which needs to manage files, we simply copy a selection of layout elements and then a simple paste get's of most of the way there. Want to know how to accomplish this efficient feat? Just watch the video and follow along with the sample file!
Comments
Signature custom function vs MD5
Hello Matt,
I happen to be working on something similar at the moment, so good timing on this video. I'm curious as to why you're using the Signature custom function instead of GetContainerAttribute ( Files::file ; "MD5" )? Is there an issue using MD5 or some advantage to using the output of the Signature custom function?
Regards,
Micah Woods
Custom Functions are good wrappers
Hey Micah. It's mostly because the MD5 algo does have some collision potential (although VERY, VERY UNLIKELY in our use here) and the Custom Function can be modified easily with distributed effect. There's nothing wrong with using the older GetContainerAttribute() it just lacks the flexibility of being able to go with a hex or base64 result.
MD5 collision potential will also increase over time as there are more and more digital files being created and as more information is stored your potential goes up.
But again, VERY UNLIKELY! Given that computational power continues to increase, and is not a big cost consideration in this use case, I just figure I'll increase my odds of less collision potential with an algo that's a bit newer.
If you're into the math, there's a good response on this StackOverflow.
https://stackoverflow.com/questions/41769724/probability-of-collision-in...
-- Matt Petrowsky - ISO FileMaker Magazine Editor
Thanks Matt, our file
Thanks Matt, our file structure uses the PK of the file record and then the MD5 and then the file, so, not worried about a collision since there will only be a few files in each folder. We're developing a server process that pushes all files to S3 (this is another advantage of using a single table for all containers, this ends up being a pretty simple process). If you're curios, here's what the url looks like:
https://some.bucket.s3-us-east-2.amazonaws.com/Containers/file/file/63780727065366-7A8CE2A2-A7D1-4B6C-8D1C-51D94FD04F24/952D2C56D0485958336747BCDD98590D/Hello.txtTable name = file
Field name = file
PK = 63780727065366-7A8CE2A2-A7D1-4B6C-8D1C-51D94FD04F24
Md5 = 952D2C56D0485958336747BCDD98590D
File = Hello.txt
There may eventually be an updated version of this file (someone makes a change to Hello.txt and replaces the file in the container field). So, then the new file gets stored here (the only change is the MD5 segment of the path):
https://some.bucket.s3-us-east-2.amazonaws.com/Containers/file/file/63780727065366-7A8CE2A2-A7D1-4B6C-8D1C-51D94FD04F24/86FB269D190D2C85F6E0468CECA42A20/Hello.txt-- Micah
Transfer photos to your system
Hey Matt,
thanks for your video! I have an application, where I use some 16'000 Passport ID photos (for a sport event). Is there a way to transfer all these photos to your system? Is it wise to do so ... and for backup reasons would you use an open storage for the container field?
Regards
Charles A. Keller
Doesn't works for me
Hi Matt,
When I add 3 files to Wedi, for example, and delete the second one, all 3 files are deleted from Wedi.