Data Management - Adjustments

[weetbix_warrior]

After any general advice anyone can offer around managing and maintaining many different adjustments I have to make to a dataset.

What I'm dealing with in the real world is SAS process (that I'm migrating to R) that has a DATA step that changes a whole bunch of values for example:
if Id is in (list of ids) then nature = "new value";
if Account_Id in (list of Ids) then limit = "new value";
if type = 'example_type' and data source in (list of sources ) then do;
amount1 = new_amount;
amount2 = new_amount;
end;

etc. etc.

The thing is there is lots of them.

I think I would rather have the adjustments stored in a database where there's an accompanying reason/comment for the adjustment, an effective date and an expiration date and some other useful info instead of just continually adding lines of code to a code file.

And then have an R process/function that reads that file/database, generates the statements or pieces of code to apply to the input dataset (checking if the adjustment is still effective).

Is this sensible? Should I just write out the individual adjustments as regular R code with # comments and keep changing updating the main R script with changes? Any other ways of managing something like this?

TIA

[nirgrahamuk]

@weetbix_warrior

And then have an R process/function that reads that file/database, generates the statements or pieces of code to apply to the input dataset (checking if the adjustment is still effective).

I vote for that. Why not ?

[rene_at_coco]

I asked a similar question before and maybe that discussion may give you an idea on how to approach your specific situation.

Identify best practices for working with changing data

What will work depends on the skills of the analysts and obtaining the cooperation of gatekeepers like the IT department. The original sin is if the csv files derive from spreadsheets, which are vulnerable to entry errors and untrapped logic. Using a relational data base management system for data entry provides the opportunity for validation to at least prevent errors like entering a literal string $1,023.10 in a numeric field. On the other hand, in addition to user training on data entry, hosting is needed that may not fit comfortably on existing servers and sooner or later a database administrator will be needed to keep the system running. Coming back to Earth, the obvious candidate is …