Cross-table interval proposer and related functionality

[tim-band]

Cross-table intervals: getting an end time given a start time in a related table.
As with same-table intervals, the start time needs to be marked with the "start" role.

Many actual changes here:

Cross-table interval proposer (so you can have the start of an interval as person.birth_datetime and any table that has a person_id can get a date relative to that). This is surprisingly useful; the proposer keeps proposing delightful things!
New "extract" proposers that can extract Date from a DateTime in the same table (very useful for OMOP), and a Year, Month or Day from Date or DateTime (useful for OMOP's person table).
Fixed an issue Yean Hoon found with her tests failing because her SRC_SCHEMA environment variable was set and the tests were not removing it.
Adjusted the colouring; it turns out that making the prompt a different colour to the user's input confuses cmd.Cmd. If we switch to the cmd2 package we should be able to get coloured prompts back.
Select gets tab completion!
Colour theme can be set from an environment variable.
New Dockerfile that's usually faster to build.
Fixed a crash in multivariate proposers (very simple fix)
Fixed a crash with multiple concept IDs in the same partitioned generator. This was really difficult to fix!
Added MSSQL to partition generator tests and found out they don't work. Big fix.

[yhong123]

Hi @tim-band , I tested the DateAfterProposer on the Pagila rental table. I first set rental_date as the anchored column and then ran the propose command on the return_date column.

I got the following sample data:

+---------------------------+--------------------------------------------------------------------+--------------------+
| source | 4. generic.anchored_provider.normal_date [anchored to rental_date] | 5. generic.datetime.datetime |
+---------------------------+--------------------------------------------------------------------+--------------------+
| 2022-07-29 11:22:10+01:00 | 1970-01-07 07:46:27.353582 | 2022-09-01 06:36:42.082114 |
| 2022-08-26 08:39:40+01:00 | 1970-01-02 12:13:09.444402 | 2022-04-11 15:53:28.046470 |
| 2022-07-15 11:29:48+01:00 | 1970-01-07 20:13:27.668529 | 2022-02-18 11:18:49.357977 |
| 2022-08-05 06:24:15+01:00 | 1970-01-01 00:00:00 | 2022-05-07 11:02:36.611369 |
| 2022-07-13 10:37:04+01:00 | 1970-01-01 00:00:00 | 2022-11-19 17:53:17.581549 |
| 2022-07-13 13:17:22+01:00 | 1970-01-01 00:00:00 | 2022-02-25 21:18:53.751723 |
| 2022-08-27 03:22:02+01:00 | 1970-01-01 00:00:00 | 2022-09-07 23:22:15.383224 |
| 2022-08-21 22:49:46+01:00 | 1970-01-03 16:56:24.846739 | 2022-06-11 23:25:31.402334 |
| 2022-07-18 04:07:18+01:00 | 1970-01-06 13:40:33.336946 | 2022-06-20 22:13:28.472823 |
| 2022-08-30 00:30:25+01:00 | 1970-01-05 02:30:45.025738 | 2022-08-23 13:42:39.555117 |
| 2022-06-27 03:31:54+01:00 | 1970-01-09 13:51:58.971780 | 2022-12-22 06:45:15.622524 |
| 2022-07-13 08:09:11+01:00 | 1970-01-01 00:00:00 | 2022-05-10 13:49:43.926202 |
| 2022-07-13 14:04:22+01:00 | 1970-01-01 00:00:00 | 2022-03-31 18:45:36.439753 |
| 2022-07-13 19:20:10+01:00 | 1970-01-01 13:41:29.580341 | 2022-11-04 13:43:44.304710 |
| 2022-08-08 12:40:32+01:00 | 1970-01-15 10:17:44.537488 | 2022-10-22 04:22:20.795851 |
| 2022-08-21 09:29:33+01:00 | 1970-01-06 23:55:39.839483 | 2022-10-19 20:22:49.066087 |
| 2022-08-24 01:06:40+01:00 | 1970-01-01 00:00:00 | 2022-02-02 21:00:34.345928 |
| 2022-06-26 13:44:11+01:00 | 1970-01-04 21:33:52.011760 | 2022-12-02 21:46:57.193264 |
| 2022-07-15 04:19:43+01:00 | 1970-01-09 16:56:10.580413 | 2022-03-24 13:46:30.689178 |
| 2022-07-11 19:13:50+01:00 | 1970-01-09 09:18:59.393509 | 2022-05-22 03:20:05.766281 |
+---------------------------+--------------------------------------------------------------------+--------------------+

The data generated by generic.anchored_provider.normal_date, anchored to rental_date, shows year 1970, which doesn’t seem quite right. Did I miss any configuration or setup step?

[tim-band]

Hi @tim-band , I tested the DateAfterProposer on the Pagila rental table. I first set rental_date as the anchored column and then ran the propose command on the return_date column.

I got the following sample data:

+---------------------------+--------------------------------------------------------------------+--------------------+ | source | 4. generic.anchored_provider.normal_date [anchored to rental_date] | 5. generic.datetime.datetime | +---------------------------+--------------------------------------------------------------------+--------------------+ | 2022-07-29 11:22:10+01:00 | 1970-01-07 07:46:27.353582 | 2022-09-01 06:36:42.082114 | | 2022-08-26 08:39:40+01:00 | 1970-01-02 12:13:09.444402 | 2022-04-11 15:53:28.046470 | | 2022-07-15 11:29:48+01:00 | 1970-01-07 20:13:27.668529 | 2022-02-18 11:18:49.357977 | | 2022-08-05 06:24:15+01:00 | 1970-01-01 00:00:00 | 2022-05-07 11:02:36.611369 | | 2022-07-13 10:37:04+01:00 | 1970-01-01 00:00:00 | 2022-11-19 17:53:17.581549 | | 2022-07-13 13:17:22+01:00 | 1970-01-01 00:00:00 | 2022-02-25 21:18:53.751723 | | 2022-08-27 03:22:02+01:00 | 1970-01-01 00:00:00 | 2022-09-07 23:22:15.383224 | | 2022-08-21 22:49:46+01:00 | 1970-01-03 16:56:24.846739 | 2022-06-11 23:25:31.402334 | | 2022-07-18 04:07:18+01:00 | 1970-01-06 13:40:33.336946 | 2022-06-20 22:13:28.472823 | | 2022-08-30 00:30:25+01:00 | 1970-01-05 02:30:45.025738 | 2022-08-23 13:42:39.555117 | | 2022-06-27 03:31:54+01:00 | 1970-01-09 13:51:58.971780 | 2022-12-22 06:45:15.622524 | | 2022-07-13 08:09:11+01:00 | 1970-01-01 00:00:00 | 2022-05-10 13:49:43.926202 | | 2022-07-13 14:04:22+01:00 | 1970-01-01 00:00:00 | 2022-03-31 18:45:36.439753 | | 2022-07-13 19:20:10+01:00 | 1970-01-01 13:41:29.580341 | 2022-11-04 13:43:44.304710 | | 2022-08-08 12:40:32+01:00 | 1970-01-15 10:17:44.537488 | 2022-10-22 04:22:20.795851 | | 2022-08-21 09:29:33+01:00 | 1970-01-06 23:55:39.839483 | 2022-10-19 20:22:49.066087 | | 2022-08-24 01:06:40+01:00 | 1970-01-01 00:00:00 | 2022-02-02 21:00:34.345928 | | 2022-06-26 13:44:11+01:00 | 1970-01-04 21:33:52.011760 | 2022-12-02 21:46:57.193264 | | 2022-07-15 04:19:43+01:00 | 1970-01-09 16:56:10.580413 | 2022-03-24 13:46:30.689178 | | 2022-07-11 19:13:50+01:00 | 1970-01-09 09:18:59.393509 | 2022-05-22 03:20:05.766281 | +---------------------------+--------------------------------------------------------------------+--------------------+

The data generated by generic.anchored_provider.normal_date, anchored to rental_date, shows year 1970, which doesn’t seem quite right. Did I miss any configuration or setup step?

I have addressed this now. Anchors are chosen at random from the anchor column in the source database. This might not always be quite right, but it's probably better than this. We still don't explain to the user what is happening with anchors though.

[yhong123]

Hi @tim-band , I tested the DateAfterProposer on the Pagila rental table. I first set rental_date as the anchored column and then ran the propose command on the return_date column.
I got the following sample data:
+---------------------------+--------------------------------------------------------------------+--------------------+ | source | 4. generic.anchored_provider.normal_date [anchored to rental_date] | 5. generic.datetime.datetime | +---------------------------+--------------------------------------------------------------------+--------------------+ | 2022-07-29 11:22:10+01:00 | 1970-01-07 07:46:27.353582 | 2022-09-01 06:36:42.082114 | | 2022-08-26 08:39:40+01:00 | 1970-01-02 12:13:09.444402 | 2022-04-11 15:53:28.046470 | | 2022-07-15 11:29:48+01:00 | 1970-01-07 20:13:27.668529 | 2022-02-18 11:18:49.357977 | | 2022-08-05 06:24:15+01:00 | 1970-01-01 00:00:00 | 2022-05-07 11:02:36.611369 | | 2022-07-13 10:37:04+01:00 | 1970-01-01 00:00:00 | 2022-11-19 17:53:17.581549 | | 2022-07-13 13:17:22+01:00 | 1970-01-01 00:00:00 | 2022-02-25 21:18:53.751723 | | 2022-08-27 03:22:02+01:00 | 1970-01-01 00:00:00 | 2022-09-07 23:22:15.383224 | | 2022-08-21 22:49:46+01:00 | 1970-01-03 16:56:24.846739 | 2022-06-11 23:25:31.402334 | | 2022-07-18 04:07:18+01:00 | 1970-01-06 13:40:33.336946 | 2022-06-20 22:13:28.472823 | | 2022-08-30 00:30:25+01:00 | 1970-01-05 02:30:45.025738 | 2022-08-23 13:42:39.555117 | | 2022-06-27 03:31:54+01:00 | 1970-01-09 13:51:58.971780 | 2022-12-22 06:45:15.622524 | | 2022-07-13 08:09:11+01:00 | 1970-01-01 00:00:00 | 2022-05-10 13:49:43.926202 | | 2022-07-13 14:04:22+01:00 | 1970-01-01 00:00:00 | 2022-03-31 18:45:36.439753 | | 2022-07-13 19:20:10+01:00 | 1970-01-01 13:41:29.580341 | 2022-11-04 13:43:44.304710 | | 2022-08-08 12:40:32+01:00 | 1970-01-15 10:17:44.537488 | 2022-10-22 04:22:20.795851 | | 2022-08-21 09:29:33+01:00 | 1970-01-06 23:55:39.839483 | 2022-10-19 20:22:49.066087 | | 2022-08-24 01:06:40+01:00 | 1970-01-01 00:00:00 | 2022-02-02 21:00:34.345928 | | 2022-06-26 13:44:11+01:00 | 1970-01-04 21:33:52.011760 | 2022-12-02 21:46:57.193264 | | 2022-07-15 04:19:43+01:00 | 1970-01-09 16:56:10.580413 | 2022-03-24 13:46:30.689178 | | 2022-07-11 19:13:50+01:00 | 1970-01-09 09:18:59.393509 | 2022-05-22 03:20:05.766281 | +---------------------------+--------------------------------------------------------------------+--------------------+
The data generated by generic.anchored_provider.normal_date, anchored to rental_date, shows year 1970, which doesn’t seem quite right. Did I miss any configuration or setup step?

I have addressed this now. Anchors are chosen at random from the anchor column in the source database. This might not always be quite right, but it's probably better than this. We still don't explain to the user what is happening with anchors though.

@tim-band I pulled the latest code and ran the propose command on the return_date column, but I’m getting the following error. It looks like there’s a timezone mismatch in the generated interval data.

datafaker/proposers/intervals.py:126 in __init__                                                      
│                                                                                                                                                                
│   123 │   │   │   return                                                                                                                                       
│   124 │   │   intervals = self.generate_intervals(400)                                                                                                         
│   125 │   │   self._fit = buckets.fit_from_values(                                                                                                             
│ ❱ 126 │   │   │   [(b - coerce_to_datetime(a)).total_seconds() for (a, b) in intervals]                                                                        │
TypeError: can't subtract offset-naive and offset-aware datetimes