How to use both AUCPR and AUC as metric for 'deeptable.fit'

[Jwenyi]

Hi there,
I'm trying to use AUCPR and AUC as metrics, so I defined the param of ModelConfig like metrics=[keras.metrics.AUC(name="AUCPR", curve='PR', num_thresholds=1000),keras.metrics.AUC(name="AUC", curve='ROC', num_thresholds=1000)] . However, it did not work and raised a error 'AttributeError: 'AUC' object has no attribute 'name''.
Besides, I would like to know how to input pre-defined validation data to DeepTable.fit. I read its source code and found no related description. Is it correct if I set validation_data = (val_x, val_y)?

I would be grateful for any suggestions!

Best,
Wenyi Jin

[oaksharks]

Hi @Jwenyi ,
Before, DeepTables accepts str or a callable function as metrics, that is why you encountered an error "'AttributeError: 'AUC' object has no attribute 'name'.", but now, it can also accept an instance of keras.metrics.Metric as the example:

DeepTables/deeptables/tests/models/deeptable_test.py

Lines 235 to 258 in d1f3999

	def test_multiple_metrics(self):
	from tensorflow import keras
	# loading data
	df = dsutils.load_bank()
	df_train, df_test = train_test_split(df, test_size=0.9, random_state=42)
	y = df_train.pop('y')
	y_test = df_test.pop('y')
	# training
	config = deeptable.ModelConfig(earlystopping_patience=5, apply_class_weight=True,
	metrics=[keras.metrics.AUC(name="AUCPR", curve='PR', num_thresholds=1000),
	keras.metrics.AUC(name="AUC", curve='ROC', num_thresholds=1000)])
	dt = deeptable.DeepTable(config=config)
	model, history = dt.fit(df_train, y, batch_size=128, epochs=10)
	# evaluation
	result = dt.evaluate(df_test, y_test, verbose=0)
	names = list(map(lambda m: m.upper(), result.keys()))
	print('score:', result)
	assert "AUCPR" in names
	assert "AUC" in names

For the second question, you can really set validation data in this way.