ART Data Limiter Issue & Resolution
ICA recently discovered and resolved a data suppression issue.
What happened? ICA discovered an entry exit data limiter that Wellsky, our software vendor, turned on in February 2020; it’s since been removed at our request. Any reports that were run between 2/13/2020 and 7/6/2020 excluded entry exit data where an exit date existed prior to 3 years from the date the report was run. Nearly 1 million entry exits in Minnesota's HMIS were suppressed during this time. Fortunately, this did not affect the System Performance Measures submission that occurred the same month.
What should I do? If you or any users you know ran reports in this time period for data older than 3 years, you should consider that data invalid and re-run the report(s). For example, if on 4/1/2020 you ran the 054 Returns to Homelessness report with a start date of 1/1/2017, any clients who exited between 1/1/2017 and 3/31/2017 would have been excluded.
The whole story
In Minnesota's HMIS, we've always had a service transaction limiter set to 3 years. We've known about this limiter since 2017 and, when we need to pull service transactions older than 3 years, we request that Wellsky lift this 3-year service transaction limiter. We have never had a limiter for entry exit data.
What is a data limiter? A data limiter suppresses data from pulling into reports based on an established parameter, in this case, date. HMIS contains a lot of historical data. When there is a high volume of data, ART database performance suffers, meaning daily builds take longer and larger reports may fail or take excessive amounts of time to run (e.g. how fast a scheduled report appears in your inbox). One way to improve speed is to shut off the flow of some of the data by archiving or reducing the volume of data constantly flowing through the database. This is a typical strategy to boost database performance.
Can you say that in plain language, please? Of course! Let's say ART database storage is like a giant library with central air conditioning. In this library, there's so many rooms and the air ducts stretch so far that the air heats before it reaches all the rooms. One way to ensure the place stays cool is to close the vents and doors to the newspaper archives and encyclopedia section, since no one goes in there anyway. The rooms are stuffy, but for the most part, it doesn't bother anyone. If someone does need access, they just need to ask, and the librarian will walk you back with a key.
What data was limited? Any Entry Exit where an exit date existed prior to 3 years from the date the report was run. This means that if a client exited prior to 3 years from report run date, the record will not appear at all. Here's a table that displays the difference in statewide datasets with and without the entry exit limiter:
Note: Service transaction data is limited differently. Service transactions with a start date prior to 3 years from the date the report was run (i.e. today's date) are suppressed. The service transaction limiter is now set to 5 years as we believe it a more appropriate duration with no noticeable performance differences.
Timeline of events
Overall, this limiter was on from 2/13/2020 - 7/6/2020. From 2/7/2020 – 2/13/2020, a 5-year limiter was in place.
February 2020 - incident: Per ICA protocol, during the System Performance Measures federal submission project, we requested that Wellsky switch the service transaction data limiter from 3 to 5 years so we could access the necessary historical data for our CoCs. Wellsky completed this request. However, when they adjusted the service transaction data limiter, they also turned on an entry exit limiter for the first time, setting it to 5 years on 2/7/2020, then adjusting it along with the service transactions to 3 years on 2/13/2020. Wellsky did not inform ICA of this new limiter. (Put differently, the Wellsky librarian discreetly closed the air vents to the whole back half of the library.)
February - May 2020: ICA noticed a small number of scenarios where data anomalies appeared, but no clear pattern emerged. It is not unusual for live databases to have constantly shifting numbers.
Early June 2020 - discovery: During an exploratory project to compare Qlik data to ART so we could better understand the limits of its use, ICA discovered what seemed initially to be an odd finding. We identified 964,030 “new” records from prior to 2017 that appeared in Qlik, but not in ART. After significant digging, we learned Wellsky placed a limiter of 3 years of entry exit data. We immediately requested its removal. Then, with the limiter off, the nearly 1 million records reappeared in ART and matched the Qlik records.
Late June 2020 - escalation & investigation: Initially, Wellsky was unable to tell us how long it had been applied to our database as they do not keep records of this application. After some pressing, they shared in the case, “So far no one remembers when this was implemented. It has been several years at least.”
Fortunately, this was not the case. By investigating all known historical data pulls in our completed queue, ICA discerned this limiter was not in place in 2017, nor as recently as August 2019.
Early July 2020 - resolution: With the unsatisfactory response from Wellsky, ICA escalated the issue to ICA leadership, who engaged Wellsky leadership. ICA requested Wellsky isolate the exact date the limiter was applied and that they commit to transparently share how and when this type of limiter is applied to a site with their stakeholders.
Wellsky located when they applied this limiter, which stemmed from a misinterpreted ICA request (see February).
Mid July 2020 – Early August – additional testing and transparent communication: ICA leadership informed Minnesota HMIS governing board on July 13th, then shared with ICA staff. Based on staff input, we then tested several additional datasets, including this year’s System Performance Measures (SPM) submission. We were able to conclusively prove that this did not impact the SPM submission. With that clarity, we’re informing key partners (if the SPMs had been impacted, we would also provide corrective actions we’d need to take along with this communication). We'll also make this information available via our next newsletter.
What we are doing to prevent a recurrence moving forward
We've pushed Wellsky to better inform their customers of this practice and change their protocol. The Minnesota team requested written protocol about when either the entry exit or services transactions limiter is used on an implementation and how customers are informed as a matter of practice. Wellsky's support team developed an internal process to confirm what limiters a site has and, when requested in a case, only change the ones currently set.
We've established a new data limiter protocol for Minnesota's HMIS. Effective 7/6/2020, Wellsky removed the EE limiter completely. We have no plans to add this limiter as we see no related performance issues. In addition, effective 7/21/2020 and per our request, Wellsky adjusted the Service Transaction limiter from 3 years to 5 years. This will help us sidestep the need to turn it on and off as most historical data requests we receive are limited to a five year window.
We're clearly documenting data limiter protocols for internal and external use through our Knowledge Base. In addition to the data limiter changes above, ICA will soon author an internal protocol document so ICA staff understand what the data limiter is and what to check for when submitting Wellsky cases regarding a historical data. We will also develop a Knowledge Base article for users. These tools should help equip our Helpdesk and users to better identify when they need historical data and how to engage ICA's Reporting & Evaluation team for assistance.
We'd like to apologize for any inconvenience this caused. If you believe a user or a project you worked on during this time was impacted by this data limiter, please contact the Helpdesk at mnhmis@icalliances.org.