Data retention CLI

[DaimonPl]

Is there any easy way to configure automatic retention for spline data? For example automatically delete entires older than 3 months ?

[wajda]

No, not yet. It could be added in the future though, it was slightly mentioned a couple of times in some local discussions.

[DaimonPl]

Ok, I think it might be useful in big setups. We have hundreds of jobs running everyday and dozens of jobs running every hour pretty sure we would need to scale arangodb quite often without retention

Any idea if retention can be somehow added in arangodb itself ?

[wajda]

Well...not in ArangoDB natively (afaik), but you could use your custom script. However this approach is kinda fragile and dirty, and I wouldn't recommend doing it. Especially in a busy environment.
To do it correctly and avoiding breaking the database consistency you need to respect the graph structure and cascade the deletion properly.

On the other hand if you could find a logical time gap in your data pipelines, big enough (e.g. one hour), you can try to cut the DB in the middle or even closer to the end of that time gap. Simply delete all documents with _created < my_timestamp from all collections excepts dataSource, all in one transaction.
This is still a hacking approach but having the idle time frame large enough it practically should not create any problems.
Also keep in mind that the described approach might not work in Spline 0.6+

[DaimonPl]

Thx for detailed answer :)

No way to find any time gap on our Hadoop it's processing 24/7/365 something all the time :)

[wajda]

Awesome. I see you case will be a really great test case for Spline :))

[pratapmmmec]

Thanks for responding. Do we have any tentative timeline for its release?

[wajda]

We'll try to implement something simple in 0.5.4 which should be out in the course of the month. We are addressing performance bottlenecks in this release, so this feature logically fits here.

[wajda]

Solution:

1. Add a command to the Admin CLI to remove the lineage data older than given date.
2. Add an option to the Gateway to do the same thing periodically based on a provided cron expression.